Born Accessible: Designing Graphic Content for Inclusive Experiences

[MUSIC PLAYING]

VOICEOVER: Born Accessible: Designing Graphic Content for Inclusive Experiences. Speakers: John Gardner, founder of ViewPlus. Julia Winter, founder and CEO of Alchemy. Arvind Satyanarayan, Associate Professor of Computer Science, MIT. Moderator: Roberto Manduchi, Professor of Computer Science and Engineering, UC Santa Cruz.

ROBERTO MANDUCHI: Thank you, Karae and Ross. This is Roberto Manduchi, speaking from the University of California, Santa Cruz. I’m a Professor of Computer Science and Engineering here, and I’m also on the Board of Directors of Vista Center. Vista is hosting Sight Tech Global, this fantastic conference that I’ve had the honor to participate in since it started in 2020. And today, here for our panel on Born Accessible. Designing Graphic Content for Inclusive Experiences, I have with me Arvind Satyanarayan from MIT, Julia Winter from Alchemy, and John Gardner for ViewPlus. And I would like to start by asking my guest to introduce herself and tell us a little bit about her background. Let’s start with Arvind. Arvind, please go ahead.

ARVIND SATYANARAYAN: Thanks, Roberto. Hi, everyone. I’m Arvind Satyanarayan. I am a Professor of Computer Science at MIT. I lead a research group that studies visualization as part of the Computer Science and Artificial Intelligence Lab, or CSAIL. And so in my research group, we study lots of different aspects of visualization, from building software tools to make interactive visualizations, apply visualizations to understand machine learning models, think about misinformation and disinformation around visualizations. And of course, the reason that I’m here, how we make visualizations is accessible to people with disabilities. Excellent.

ROBERTO MANDUCHI: Julia, tell us something about yourself.

JULIA WINTER: Hi, I’m Julia Winter, CEO and Founder of Alchemy, based outside of Detroit. And before that, I was a 20-year high school chemistry teacher. Started the company to build interactive learning tools for chemistry and other STEM subjects. And came across the really gnarly problem of making them directly accessible. So we wanted to do that. So I love the title, Born Accessible, and I will say, I have watched Sight Tech Global for the last three years, and my team likes to book clubs about, like, we’ll divide up the talks. And so I’m so honored to be part of this panel.

ROBERTO MANDUCHI: This is great. We are honored to have you in the panel. Now, last but not least, John.

JOHN GARDNER: Hi, I’m John Gardner. I’m a-I’m blind. I lost my sight about 35 years ago, rather suddenly. And at the time, I was professor of physics at Oregon State University. And I found it was hard to continue doing my research because it required a lot of graphical analysis and graphics were not accessible to blind people. They barely are today. I founded a spinoff company called ViewPlus, about a little over 25 years ago. And it’s grown into a major company. It makes tactile graphics embossers. It’s the only company that makes tactile graphics embossers. In fact, we sell them all over the world. And they have software that goes with it that we will probably run across later in this program. And I appreciate being here. I admire the work that SciTech does. SciTech Global has done. And I’m very happy to be on the panel. Thanks.

ROBERTO MANDUCHI: Excellent. And I’m sure that many of our listeners are very familiar with ViewPlus and this product. I would hope so. Well, my first question in this panel is to John, in fact. So, John, when people make graphs or images or maps, the way, the current way to make them accessible works in this way: a sighted person creates their image, a graph, a map. And then when they remember, they will add some alt text, which is some text that describes what is in there. What could go wrong? Well, here is the thing. Lots of things, of course, could go wrong. How can we go beyond this paradigm? A sighted person adding at their will some alt text, if they remember. And how can you create documents that are natively accessible? Documents that can be consumed in more ways than just visually from the beginning. What do you think?

JOHN GARDNER: Well, we’re trying to solve that problem. We being a group called Inclusio, which is funded in a large way by the National Science Foundation. ViewPlus is the commercial lead of that project. And that’s precise. And actually, our goal is to be able to make mainstream graphics that are automatically accessible. And we can talk later about how that is done. But I think it is certainly possible. And we’re hoping to make it easy to simply make sure that when something is published as a graph, it will be accessible.

ROBERTO MANDUCHI: Well, you made us curious, John. Tell us something more. How do you do this magic?

JOHN GARDNER: Well, we put some Meta software into a file. At the present time, the file format pretty much has to be scalable vector graphics, SVG. But there are other formats that will probably come along in the future that we’ll work on. And what this does is makes it, what is it? What’s the word? Interoperable among technologies. I mean, for example, my favorite way of reading graphics is by audio visual, where you have a tactile copy. Nice tactile copy, of course, made on one of our embossers. And you touch it, and it tells you what it is. For example, you can; I cannot imagine a verbal description of the United States that really tells me what it is. Or what the map of the United States looks like. But as soon as I have a tactile copy, I can feel it. But I’m not very good, and most blind people are not very good at reading tactile graphics. So the tactile graphic talks. And if I put my finger on California, it should say ‘California’. If I put it on Oregon, it should say ‘Oregon’. Et cetera, et cetera. And there can be other information. And in fact, you could put a whole book into a graphic. That’s called audio tactile graphics. One of the collaborators in this group is a small company called Vital that makes a haptic, audio haptic access in which you feel something, and the screen vibrates. So you could feel this map of the United States. And as you cross a line, it would vibrate so that you know you’re going across a state line. And it also, if you tap on it, will speak the name of the state, et cetera. And a third way is to simply make a free standing tactile graphic in which there is text. The text is Braille. Braille labels are usually too big to put onto a tactile graphic. So what you do is you put small mnemonics of one or two Braille characters. And then you make a book. Telling what the labels that go with those are. Which is a clumsy way to read something, but it works. And the present time, we’re less than a year into this project, phase two project. We have it working for all three of those methods. So we can make a scalable vector graphic. And I can choose which of those three I want to use. Push a button. I get a Braille copy. Or I get a. A tactile copy intended for audio tactile, et cetera, et cetera. Now this is a mainstream, can be a mainstream SVG. Published in a professional journal. And there’s no reason that I can’t just look at that and extract any one of these three methods. And those are only the first three. Other methods will be coming along quite rapidly. It’s intended to be very easily expandable. There’s a new couple of new multi-line graphics tablets that are on the market. They’re not cheap. But they do work. And it’s fairly straightforward to extend this so that that’s the fourth. And then there is a fifth. So there are many ways that you can consume this. That’s what I would call accessible.

ROBERTO MANDUCHI: Beautiful. And I would call it actually born accessible. So you have all of these different ways to access the data already in the document. Beautiful. Arvind, you also have worked on creating documents, so to say, that can be consumed in multiple ways. Tell us something about it.

ARVIND SATYANARAYAN: Yeah, absolutely. So one of the things that I really liked about what, you know, all of the different options that John was describing is, you know, not only do you have those different modalities available to you, right, tactile, sound, you know, verbalized speech, haptics, and things like that. But that you’re using those modalities to get different kinds of information, right? That might be information at different levels of detail. Information at different levels of resolution. Or even totally different information whatsoever. And so that’s a lot of the research that my group has been focused on. We sort of call it multimodal data representations rather than just data visualizations, if you will. And a lot of this work has, you know, been led by my Ph. D. student, Jonathan Zong, who will soon be a faculty member at CU Boulder. And in particular, one of the things that we found is, you know, as we’ve explored tactile sonified data representations, right, which is using a tone to sort of depict, you know, time series data or something like that. So higher pitches indicate, you know, higher data values. Lower pitches indicate lower data values. And then, you know, verbalized or textual representations. Is that there can be a lot of value in that. You know, in combining them together, right? So hearing just a sound play out is a really nice way of getting like a very high-level overview of trends or things like that, right? We can pick out very easily if something suddenly spikes because it goes from like, you know, a relatively flat tone to like a sudden high pitch or a sudden trough or something like that. With text, I get really, really fine-grained information, right? I can get very precise values read back to me. And things like that. And with tactile, I’m much more in control of when I’m receiving what kinds of data, right? Because I’m able to sort of move my hands around and feel the various tactile symbols. And so in that way, a lot of our research is figuring out, you know, when do we care about which modality? And how do we use those different modalities to surface what kinds of information and things like that?

ROBERTO MANDUCHI: Beautiful. That’s a great model. And with an idea that different modalities are kind of like different purposes and they can be used and interacted with in different ways. Fantastic. Exactly. Okay. Very nice. Very nice. These are all very promising, beautiful projects I’m hearing. Let me now direct towards education because, of course, we’re talking about graphic content. Let me think a little bit about STEM students, right? Science, technology, engineering, mathematics. And it is very difficult to teach STEM or to learn STEM just based on text, right? We need figures. We need graphs. We need diagrams. We need explanations. We need to somehow learn the spatial relationship of things, of atoms in the molecules, for example. Well, that’s why I’m immediately thinking about Julia. So, Julia. We need students to learn shapes and spatial relations. And clearly, if a student is a slow vision or is blind, just adding alt text is not going to cut it. It’s not going to work. So, tell us something, some possible creative solutions, some of which you have been working on, to let the students access and manipulate shapes and to learn spatial relations.

JULIA WINTER: Yeah. So, this has been ongoing projects funded by Department of Ed, NIH, NSF, and NIDILRR through SBIR programs. And we started with manipulatives that I used in my class, in my high school class. Like I would put atoms on the board. And I never taught a low vision student in my school. But these were, these, I said, why? Could we do something with these manipulatives? Manipulatives for Lewis structures is one of those things that people think of in chemistry. And could we use computer vision to recognize the pieces and then give guidance and feedback as the students learn? So, that was our very first idea in 20/20, which got funded by the Department of Education. We had no idea how to do it. We had zero knowledge. So, we figured out how to do it. And got a Phase II grant from the Department of Education. And as we built these manipulatives, we 3D printed them. We designed them. We tested them with blind students and their teachers of visually impaired, TBIs. And we added Braille. And we did research studies. And then when we were working with the teachers of visually impaired, they said, this is great. Chemistry is wonderful. But you have to get something earlier. You have to get it into first, second, third grade. Because we lose our kids way before chemistry. So, we started working with the same idea with common manipulatives in math. And we added textures to those common manipulatives to indicate color. So, when the system would say ‘pick up the brown rod’, the blind students would know what brown felt like. And so, every student can use the tools. It’s all designed to be born accessible but also inclusive. So, these tools are designed to be used in classrooms, in mainstream classrooms, but also to support the spatial relationship building that has to go on for the blind students to move into STEM. Because spatial relationships are key to success in STEM. So, we have that. And then we have a digital component. Purely digital component. Which generates dynamic alt text as the students use our keyboard-accessible control panel in our software. So, we are building for organic chemistry right now. General chemistry. And then hopefully our NIH Phase Two will come through and we’ll have K-8 math over the next two years.

ROBERTO MANDUCHI: Very nice. And I hear things like manipulatives and tactile. Which is important, right? Because by acting on things moving together. And I must say this reminds me some of the work of Professor Arthur Kashmer. Some of the people in the audience might have known him. I’m sure that John remembers him. He passed several years ago. But he also was a big proponent of manipulative ways to learn, for example, mathematics. To put together little formulas with pieces of wood, et cetera. So, I really appreciate that you’re working on that. Arvind?

ARVIND SATYANARAYAN: Can I add something? So, one thing I really liked in what Julia was describing was the notion of dynamic alt text. Right? I think one of the reasons that, you know, alt text maybe gets a bad rap is because so far we’ve treated text as a very static medium. Right? It is essentially when the person who usually the sighted person designs the text. You know, you hope they remember to provide the alt text. And then the alt text lives forever with that graphic. Right? And the alt text can only say the one thing. And what I really like about what Julia is describing is the fact that that alt text can, you know, change over time. Right? Based on the ways that in Julia’s case I think it’s STEM students. Right? Are interacting and manipulating the manipulatives. And I think that, you know, again, from a research perspective. One of the things that gets me really excited is that this feels like a really fertile sort of like design space. Particularly right now with, you know, generative AI and large language models. Not just to be like, well, let’s throw gen AI at this problem because that’s the hot cool thing right now. But because many of those models are about manipulating text. Right? And making text this much more dynamic medium. So, one example from our own research, for instance. Again, sort of shifting back to visualizations just for a second. Is that it’s very common, you know, to see best practices for alt text around visualization to say something like, you know, this is a line chart with, you know, dates along the X axis and price along the Y axis. And, you know, maybe summarize what the trend is. And I think that’s a reasonable first place to start. Right? But, boy, is that a very frustrating piece of alt text. Because it tells you like one thing. Right? And one thing not in a very rich way. Versus for sighted people when we’re reading charts, there’s lots of things that we’re reading into it. Right? We’re reading context. We’re reading, we’re bringing our own sort of like domain information and things like that. And so, one of the things my students and I have been exploring is how we might use a large language model to bring some of that contextual information into a piece of dynamic alt text. So, for instance, imagine that you have, you know. A line chart of, you know, the price of some commodity over time. And, you know, you have essentially the large language model split the alt text into different segments to say, for instance, like, you know, between 2008 and 2010, it was the Great Recession. Right? And the prices of these commodities really fell down as a result. And then, you know, in 2020, again. Due to the COVID pandemic. Right? The prices experienced a rapid decline. And so, that’s the sort of way that I think some of these more modern technologies can help us bring these more sort of like contextual domain-specific pieces of information. And alleviate the burden on sort of the graphic designers and authors to have to remember, you know, first of all, to write the alt text. And then to convey all of this sort of rich metadata. And cram it all into just a single page. And a piece of alt text.

JULIA WINTER: You know, I will say one of the things that our users, our screen reader users who tested our technology really, really liked was once they learned how the alt text would be describing what they were using on the screen, it was always the same. So, one of the big problems with alt text is the standardization problem. Because everybody authors alt text. There are best practices. But they’re all different. And the other really interesting problem that we have looked at as an assessment company is that our tools assess student learning. The alt text can’t give it away, so you have to actually build in the learning design into the dynamic alt text. So, if you have a learning objective that wants the student to come up with X, you can’t tell them what X is. It’s similar to I look at some of your work, Arvin, and you want. Like if a student was reading a graph or a piece of data, you want them to be able to synthesize that and not just be told what the trends are. Exactly. And that is one of the really tough problems in education space because we want to build the scaffolds to allow the students to use critical thinking via alternatives to visualizations. Which it is a design problem. And really interesting to do.

ROBERTO MANDUCHI: Beautiful. Very, very interesting. And I like this concept on alt text not necessarily being just a piece of static thing. Because any graphical content can be discovered in so many ways. Right? And so, perfect. Beautiful. Well, this kind of leads me to my third main question for discussion. This to Arvin. So, all right. Again, back to the paradigm. Okay. So, we have a sighted designer, created a graph, and then as an afterthought, perhaps, adding some making accessible. Right? So, in this paradigm, blind or visually impaired individuals really are passive consumers. Here I’m standing here hoping that this guy remembered to add something so I can understand and read what this figure is about because there’s a lot of reference to this figure in this article. But I have no idea what that is. So, is it possible to turn this paradigm around and empower, I like this strong word, right? Empower blind and visually impaired individuals to create their own graphical material in a way that, of course, because they must make it accessible, is it natively accessible? Is it a pipe dream? Is it possible? Can we someday reach that? What do you think? Yeah.

ARVIND SATYANARAYAN: I think that’s a really important question around empowerment. Right? Because I think the issue is not just the one that you were describing, Roberto, where, you know, maybe the sighted designer forgets to include alt text. But even in the best, the ideal case, right, imagine the sighted author was to provide the alt text. Well, our, you know, blind or low vision reader, like, all they can do is accept or reject whatever the alt text says. Right? So that’s not particularly empowering, at least not as empowering in terms of, you know, being equitable to the sorts of reading experiences that sighted people have. Right? If I read a chart, I can accept what the journalist is saying with the chart or I can read my own meaning into the chart. Right? And so that’s definitely a big thread of research in my group. But sort of looking at the last portion of that question you were asking. As I mentioned, my student, Jonathan Zong, who’s been leading this work, he recently developed a system called Umwelt, U-M-W-E-L-T. And what Umwelt is trying to do is, you know, rather than start with the assumption that there exists a, you know, visual representation that we want to make accessible to people who are blind or low vision. Umwelt essentially is a screen reader accessible tool. Right? That is designed for screen reader users to use directly to author their own data representations. Right? So we break that assumption that there exists a graphic representation that ought to be made accessible. And instead, kind of like the title of this panel, what are the data representations that screen reader users sort of can make that are born accessible by default? And so what Umwelt gives people access to is, you know, all the modalities we’ve been talking about so far. So sound, text, and, you know, I hope to chat with Jon after this panel because we’ve just added some new functionality or we will be soon around tactile graphics. And the goal is through a fairly simple, you know, dropdown style interface, people can, you know, map, you know, different fields in their dataset to, you know, these different modalities. So you can say, you know, I want to show price as pitch. I want the text to include, you know, descriptions of, you know, these other data fields and so forth. And then sort of like the operations Julia was talking about with the manipulatives, you can move between these different modalities. So I could play the sonification, you know, hear that sudden spike in the pitch, pause it, switch to the textual representation and figure out, like, what’s the precise data value, you know, on what date did that occur and things like that. Right? And that’s a, that’s a born. Umwelt. I think, you know, there’s a lot of things that are very, very useful in the digital space. So if you want to talk about something that’s more accessible, if you will, representation because there never existed a visualization first, right there, it didn’t require a sighted person to create a visualization first and then make accessible.

ROBERTO MANDUCHI: It’s beautiful. And, and, and, then John, these kinds of things bring us back to your big, inclusive project, sounds something similar, right, to where the point where we don’t start with the priority of the visual representation of whatever this project is. want to represent, but we just offer a multiplicity of ways to to use it.

JOHN GARDNER: Well, yes, and no. Um, I will admit that that my vision is more uh, more aligned with having a visual representation that is made accessible, probably because I was a sighted scientist for so many years and uh, wrote so many papers. And now, thinking about the graphs that I made for them um, gosh, I should have done a lot better than what what I published. My graphics were very dense and full of full of information and full of too much information actually. But there are certainly ways that a blind person can make information that is simultaneously visually accessible and um, and accessible in various other ways, and in a sense, that’s sort of what I have taken on as as a challenge because because we’re really trying to make things that work for everybody. And I, I would like to mention Harvin talked about using tone uh, as a way to measure value uh. The very first product that the ViewPlus put on the market back in year 2000 was an audio graphing calculator that did exactly That it played a graph by um by tones and graph went up the tone phase frequency went up, but it wasn’t just the tone. You could stop it and have it give you the data points. It could do things like play to a minimum and speak the data at the minimum. It was quite flexible. It turns out that in testing this, we just didn’t have enough blind people to test it on. So we tested it on several large classes at Oregon State and we found quite surprisingly, it took very little training for somebody to be able to use a tone graph and to basically use it perfectly to know exactly what that graph looked like. Another tone use that I’m very pleased with are GIS images. You have a map, you have the population of states, counties, whatever. If they’re well-made, they’re in a color and the color gets brighter and lighter and darker depending on the intensity. Again, back to the map of the United States. California is the most popular state. It would have a very bright color. Wyoming would have a dull color. In tactile graphics, you can do that by making the size of the dots. California has big, tall dots and Wyoming has little small dots. But then if you can run your finger across it and hear the tone playing, that’s what I call really good access. And that exists already. We’ve been doing that for years. Absolutely.

ROBERTO MANDUCHI: Well, this was great. I think we had a fantastic discussion. I want to thank again my guest, Arvind Satyanarayan from MIT, Julia Winter for Alchemy, John Gardner from ViewPlus. This was the Born Accessible, Designing Graphic Content for Inclusive Experiences. This is the host, Roberto Manduchi. Back to Karae and to Vista Center for Sight Tech Global. Thank you, everyone.

JULIA WINTER: Thank you.

ROBERTO MANDUCHI: Thank you.

[MUSIC PLAYING]