The Dynamic Tactile Device: That "Holy Braille" for education is near

Following up on last years' discussion of the APH and Humanware collaboration to create an education-focused tactile display (see next session), Greg Stilson updates Sight Tech Global on the project's progress and APH's work toward an SDK for developers to build on the tactile display. Greg Stilson will also lead a breakout session for attendees who want to go deeper on the Dynamic Tactile Device.

[MUSIC PLAYING]

BRIAN: The Dynamic Tactile Device– That Holy Braille for Education is Near. Moderator– Devin Coldewey. Speaker– Greg Stilson.

DEVIN COLDEWEY: Thank you, and thank you, Greg, for joining us here today. I think that probably a lot of our viewers are familiar with APH, but they may not be familiar with your role there and how you apply technology to the APH mission. So maybe we can just start out by you explaining what it is you do there and how that helps people.

GREG STILSON: Sure. Thanks for having me. So my name is Greg Stilson. I run the Global Technology Innovation team at the American Printing House for the Blind. And what this means is it’s a group that is a combination of technical product managers, quality assurance analysts, and software engineers, with the mission to develop technology, or partner with companies to develop technology, that serves our core mission, which is the American Printing House for the Blind is primarily focused in the education realm. We are a federally funded organization in the United States to basically provide technology for– to advance the education of students who are blind or have low vision. We started out primarily in the education sector but have grown significantly really in the last 10 years, more towards an international entity now, providing products all over the world. We also partner with different organizations to distribute our products, things like the Mantis Q40 or the Chameleon. So yeah, we started off in education, and that’s really where our key focus is. That’s where all of our federal dollars go. But we’ve kind of grown beyond that to really focusing on helping anyone who is blind or low vision in any aspect of their life.

DEVIN COLDEWEY: And you focus on the technological side of things, reaching out and working with partners to develop new technologies or adapt to them.

GREG STILSON: You got it, yeah. So internally, we have software engineers who develop technology or develop applications to either go on the technology we’re building or just standard mobile applications and things like that. A lot of the high-tech hardware that we release is in partnership with other organizations, the folks like the human wears or the sparrows or the dots around the world, and so basically modifying or coming up with new technology with those organizations based on a need that we see primarily in the education system. But now, as I mentioned, it’s going beyond that as well.

DEVIN COLDEWEY: And I know that there’s been a lot of advancement in technology over the last 10, 15 years in accessibility in just general data accessibility and portability, all these things that have helped a lot of people connect with the things they need. Where have these private companies and services fallen short, in your opinion, that you’ve sort of felt the need to step in?

GREG STILSON: So it’s a good question. The first thing we often look for is where is the unmet needs? I’ll give you an example. One of the products we released a couple of years ago was the Mantis Q40. What we were hearing from folks– there was always the refreshable braille display market that used the 6-dot Perkins keyboard to input into a braille display when it’s connected to whatever other devices you want to be connected with. However, we started hearing from especially the high school kids, the transition kids, and then ultimately the folks in the workplace that they were finding that they preferred to use a QWERTY keyboard. And so being able to blend a refreshable braille display with a QWERTY keyboard that has some level of intelligence was a hole that we looked at filling at that point. And so that’s where the Mantis Q40– the idea generated from, and it kind of grew from there. So as we look forward, we’ve always been on this mission for a multi-line refresh braille display. And today, all of the braille displays that you see are single-line braille displays, which is valuable to have braille at your fingertips. But in the process of learning, especially learning content like science, technology, and math, it’s really hard to do math one line at a time.

DEVIN COLDEWEY: No kidding. So tell me a little more about this new project. Maybe you can give us a quick summary of what it actually is, and then we can talk a little more about why it’s needed before we get into the details.

GREG STILSON: Yeah, absolutely. So we previewed this a little bit last year at Sight Tech. And it is the primary goal of this project is to create a self-contained all-in-one education tool that has multiple lines of braille on it, and is also capable of rendering a tactile graphic as well. And really, what we’ve seen in the past is we’ve seen one or the other. We’ve seen products like the Canute, which are strictly single lines of braille stacked on top of each other. We’ve seen products that APH have worked on in the past, such as the Graphiti project, which was 100% a tactile graphics display. What we’re trying to do with this product, and it’s a combination of, I say, I think, the marriage of good timing and good technology coming together, is create a situation where you’re able to produce standard braille and graphics on the same surface. And there’s a huge benefit with that. What we found with the Graphiti project, for example, was, if you just give a blind person a tactical graphic with zero labels or zero context, it is really, really hard to understand what that is. And so one of the biggest pieces of tactile graphics is the labeling, is the creation of context. And you really can’t do that without providing these braille labels and things like that. So that’s primarily the function. The device is a self-contained tool with a multi-line tactile array. But it has its own braille keyboard on it to control and do input into the device as well. It’ll have your standard tablet functionality, so your Wi-Fi, your Bluetooth, your microphone inputs, headphones, all that kind of stuff. It’s also going to have an HDMI out as well so that a teacher who may not know braille can actually see what the student is doing on a monitor as well. So that’s really where we were looking to go.

DEVIN COLDEWEY: It’s interesting, providing that context. And talking about learning something like math or chemistry using a single-line braille display sounds incredibly difficult. Can you tell me a little more about what the state of the art is for a student who is in high school or something like that and is– what is the process for creating and distributing these textbooks?

GREG STILSON: That is a really good question. And I’m really sad to say that in the 20 years that I’ve been out of school as a blind student myself, we have not seen much of an evolution from a textbook distribution process, from a materials distribution process. So APH is one of the largest textbook providers here in the United States. And basically, the process goes something like this. We receive the textbook from the publisher, and it goes into a repository called the NIMAC. This special accessible file basically then goes to our braille transcription team. This transcription team will run it through some automated software to do the base level conversion into braille. But at that point, when you do that base level conversion into braille, you lose any type of proper formatting. And with subjects like science and math, formatting, especially with things like matrices, long division, the vertical formatting is crucial at that point. And so all of that is wiped out when you run it through these automated tools. And so you have a braille transcriber, whose job it is to look at the print textbook and essentially determine context into what the formatting should look like so that it makes sense to the blind reader. At some point during this entire process, we recognize that there are tactile graphics that have to get made. So we send those to our tactile graphics team, which makes them in parallel to the transcriber going through their process. After it’s all said and done, and the transcriber feels good about their side, the graphics artist feels good about their side, they all come back together. And in some cases, those graphics get combined with the text content of the book. In other cases, especially in higher-level math and things like that, you will end up with volumes and volumes of the text content and then maybe a volume of just tactile graphics. Just to put it in perspective, we did a algebra two book last year. That book took over 40– it consisted of over 40 volumes of braille and cost over $30,000 to produce. So that just gives you the context of where we are today, but also, ultimately, where we’ve really been over the last 30 to 40 years when you look at it.

DEVIN COLDEWEY: So how do you know that it’s time to do something new? How do you know that this technology is ready to go? So I know that you’re working with Dot and we’re actually talking with Dot later today about some of the technical aspects of their new braille Dot technology. But like, how do you know– because this has been– so many people have tried and succeeded to various levels with over 10, 20 years, but like you said, it hasn’t gotten that adoption. It hasn’t gotten to that level. So how did you know that this is the time to do it? And how did you start thinking about developing that into a product?

GREG STILSON: Yeah, it’s because we’ve made several attempts at this, and I will say we’re not the only ones. There’s been startups out there who believe they’ve cracked the code and were ready to release something, and it never came to fruition. So I think it’s really three points. I think the technology that Dot has created with their new braille, their braille Dot technology that they’ve created, the braille cells, is a big piece of the puzzle. We’ve never seen anything that is, number one, this durable, and number two, feels like a natural braille cell. Usually, you have to make very significant trade-offs because, at the same time, we’re also trying to reduce the cost of the braille cell, which has been astronomically high for 40, 50 years. So we look at that. So you have this emerging technology from Dot that has been proven. We’ve seen a lot when we put out our requests for information. Secondly, I think it’s a match of good partners as well. This is not something that APH is doing on our own. It’s not something that Dot’s doing on their own. So this project initially kicked off with the partnership between the American Printing House for the Blind and HumanWare, a company out of Canada. And HumanWare really has taken on the manufacturing side of things. So when you look at our different roles in this partnership, APH is really heavily focused on the UX. We’re focused on the testing. We’re focused on the software interface and design. HumanWare is doing a lot of the coding. Their engineers are actually doing the vast majority of the coding. Our team is there to assist. And Dot is providing that braille cell technology. So we each have our own roles in this partnership. So I mentioned that it’s the partnership. It’s the technology. And I think it’s just the timing as well. We’re to a point now where electronic delivery of books is not a foreign concept. It isn’t unfortunately in our field. But in the broader landscape, it is– [LAUGHS] I’ve questioned what the ratio is for reading etextbooks versus– or, I’m sorry– ebooks versus regular books now. And it’s not something that is a foreign concept. So I think, with those three pieces all coming together, and we have an internal joke of at APH that the stars and planets have to align for this project to work successfully. And we knew immediately this was not something we could do on our own.

DEVIN COLDEWEY: Roger, yeah. So let’s talk about the actual device and the development of it. I’m always really curious about this process of creating a new device with a very specific use case, especially one that has been tried before. So maybe, could you tell me a little bit about going into this project, when you realized, like, OK, the stars and the planets have aligned, we think? So what were your must-haves when you were making this, and how did you design around those?

GREG STILSON: It’s a great question. So number one, the braille– we had to find a technology that made the braille feel natural. Number one, you’re going to be teaching entirely new concepts with a product like this that’s never been done before, so teaching somebody to feel a different feeling braille dot. Or it would almost be like– I equate it to a sighted person saying, you can only read in this print font from now on. And so number one, was making sure that the braille dot felt natural and that it was durable because, being in the education system, kids are not– [LAUGHS] they’re not gentle with these products, so making sure that it was durable. So that was number one, is that, in order to do this, we had to be able to ensure that the dot felt natural. Number two was be able to find a technology that could put graphics and Braille on the same surface. When we looked at this, in order to achieve our, what I’m calling our North Star goal, that is that we can put a textbook, we can digitally deliver textbooks to students. And to do that, you have to be able to put Braille text and graphics on the same surface. That’s how it’s displayed in textbooks today, and that’s really, like I said, how you create that contextual understanding for a tactile reader.

DEVIN COLDEWEY: Yeah.

GREG STILSON: So yeah, those are sort of our top three things. But this project, as I mentioned in the past, it doesn’t move forward unless we can achieve that textbook goal. So we have all these other goals of putting a multi-line word processor on this or a graphing calculator. And all those things are in the plans, and we’re going we’re working on those as we speak. But having said all that, the funding perspective doesn’t happen unless we achieve that textbook goal. And so that’s really where we’re laser-focused at this point.

DEVIN COLDEWEY: Yeah. And I want to talk about that too. But before we get to like how it’s funded and how you’ll get it into the hands of students, I wanted to ask a little more about the development of the hardware, because this has been in the works for a couple of years, but I can’t imagine a more difficult time to attempt to design, build, and manufacture a totally new device with a pandemic, with a community that is scattered all over the world. How did you do it? And do you have any, like, do you have any advice for anybody trying to a new hardware startup like this. I’ve heard it’s unbelievably difficult.

GREG STILSON: It is. I don’t want to do UX testing on a physical device that people have to be touching virtually, I don’t want to do that again. That was really hard. So the number one thing when we were looking at the hardware was what is the right size for the tactile array? And I’m telling you, Devin, we had so many competing opinions, even internally here. We were looking at sizes, such as 20 lines by– or I’m sorry, 13 lines by 20 cells. We were looking at 28 cell lines, 30 cell lines, 32 cell lines. And the one that we all thought was going to be the winner was the 40-character line. And the reason being is because the standard piece of Braille paper today is 40 characters wide by 25 lines. So we all were thinking, before we started the testing, that, well, it’s just natural that it would be a 40-character line because that would make our software side really easy. We don’t have to modify anything that’s there. We can take the existing books and just slap them on this display and break it up vertically, and it would work just fine. And then we put it in front of people. And because the way that our tactile array works, and we have more dots than actual Braille cells to create that equidistant pins from each other to create those graphics, 40 characters wide by– we tested 40 characters by 10 lines– that was a gigantic device. That was bigger than the kid’s desk when we actually put it down on the table. And so what we ended up doing, we were doing the vast majority of this testing towards the end of 2020 beginning of 2021, is we were sending out literally woodblocks of the dimensions of what this product would be. We also sent out 3D prints. But most importantly, what we ended up doing, and I give massive credit to our tactile graphics team, is we sent out, basically, paper packets cut into the size of these tactile graphic arrays. And what we did is we put various graphics, combinations of Braille and graphics, on these different sized pieces of paper to say, imagine that you are looking at a graphic, and you can only see this much of that graphic at this time. How much are you able to understand? And we were also doing things like shrinking or scaling the graphic to the various sizes to see how much detail was lost at that point. And then what we would do is jump on Zoom and talk through. We’d make them go through the different packets item by item, graphic by graphic, and document what they were doing. Because this wasn’t something that we could just be, like, here’s a whole packet of paper, and then fill out a survey. It didn’t work like that. So when it was all said and done, I think the most important decision that we had to make in this project by sheer, I don’t know, force of will that we went through here, was we settled on a tactile array size of 10 lines by 32 cells of Braille. And where we are today, looking at the product, it 100% feels like the right move, and it was very clear by all of the feedback that we got that 32 by 10 was the preferred size.

DEVIN COLDEWEY: That’s interesting. Because you can’t figure that out just in the lab or on paper. It has to be with people who are actually using it, saying, like, oh, no, this is a no going. Oh, that’s interesting.

GREG STILSON: Not only that, but we were looking at, I mean, we were testing with teachers of the visually impaired. We were testing with students, we were testing with students with multiple disabilities. And then parents, right? What would work at the home as well as in the classroom? Because at home, you have a big kitchen table that maybe you could slap a bigger device on. But then at school, you have these little desks that they won’t fit on.

DEVIN COLDEWEY: I remember. I feel like desks should have gotten bigger sometime in the last 20 or 30 years, but probably, they got smaller. So tell me– let’s talk about the software side a little bit too. Just because I know that when I talked with Dot a while ago when they were first talking about their technology, it was very much focused on iOS and using some of the stacks in iOS to directly put graphics on it and stuff like that. And I’ll talk with them about that more, but I understand you’re working on an Android base and sort of designing your own software base from the beginning. What made you do that, and what’s the advantage that that gives you?

GREG STILSON: So we looked at the approach that Dot and others have taken in that the device is a separate peripheral for another tool, right? And the challenge that we’re facing there is that in a classroom, number one, there’s very little consistency throughout the United States on what devices these students have access to. Some districts are Chromebook districts. Some districts are iPad districts. Many districts don’t allow the student to even have the phone in the classroom. So immediately, when we were looking at this saying, all right, if this has got to be used at the same time that the sighted peers are using this, it has to be a tool that can be immediately picked up and used by the student. So creating a secondary tool to some other device was out of the question at the beginning. And secondly, the primary function of this was an ebook reader, to be a textbook reader, to be able to allow a student to immediately turn to chapter 5 or page 157 and just jump right there. Today, you use tools like a Kindle or an iPad or things like that to access your books. Those books are not connected to something else. And so for a student to, in the same fashion that they pull a book out of their backpack, we wanted them to have the exact same experience in pulling this device out of the backpack with nothing else at that stage. And so it was pretty clear that it had to be a self-contained tool, which we recognize that increases the cost, it increases– it vastly increases the development effort on our side. But it was the right move, especially with our focus on the education-side of things and making sure that those textbooks were immediate.

DEVIN COLDEWEY: Gotcha. It has to be do it right just ’cause if you’re planning on building a platform for the next few years or 10 years of essentially Braille textbooks and used by this community, do it right the first time so you don’t have to come back in two years and be like, OK, so now, we have to redo it but not as a display device for a second device.

GREG STILSON: You got it. And the benefit here is that we’re not just limited to the stuff that we do. Just because we put our own apps and experiences on it, one of the apps that we’re specifically looking at is what’s called a tactile monitor. And that’s essentially you doing the same things that Dot and other folks are doing in being able to turn this device into a receiving tool from other devices, in the same way that you can connect to a refreshable Braille display today to a smartphone or a computer and have the screen reader drive that experience. So we’re not just limited on that. The way we look at it is, it’s the best of both worlds.

DEVIN COLDEWEY: Right. Yeah. It’s not that it’s one or the other. It’s you adding on top. So you mentioned that increases the price, obviously, and also that you have these different districts and capabilities across the country and around the world. As you said, you’re increasingly an international organization. How do you get these into the hands of kids? And how does that funding work, because I know it’s a first-time device. It’s going to be expensive. There’s support. There’s a lot to think about. And so you can tell me about how this is paid for by districts and by APH. I’m just curious about the sort of financial side here.

GREG STILSON: Sure. So I can talk to you about how it works in the United States. And we’re still– we’re working with our–

DEVIN COLDEWEY: We’ll talk about the rest of the world later, next year. It’s OK.

GREG STILSON: Well, we’re trying to figure that part out as we speak. But at least here in the United States, APH is funded by a federal appropriation that’s been around since, I want to say, 1879 or something like that. It is still drastically an underfunded appropriation. Basically what we do is we look at the census that says how many students are qualified under the legal definition of blindness or low vision. And you look at– I want to say that we published it this year. It was just over 55,000 kids in the United States that qualify for services from APH. The federal appropriation gives– approximately, it comes out to be about $460 per kid to purchase specialized tools or materials or manipulatives or anything. Like I said before, APH, we’re talking about the high-tech stuff, but we also do extremely low-tech stuff, selling things like Braille paper and tactile graphics and that kind of stuff as well. So it all comes down to how many kids. So a state like California, for example, gets a bigger pot of money because they have more kids that qualify for this funding. That funding can only be spent at the American Printing House for the Blind. I always joke that it’s a very effective monopoly. At that stage, you’re looking at $460 per kid, right? So first off, for a blind kid that needs a refreshable Braille display today, which is like $2,000, you basically have to borrow from the kids that need I would say less expensive products. If, let’s say, some kid is just under the legal definition of low vision, they may not need nearly as many adaptations or tools as a student who is multiply disabled or was born totally blind, something like that. So long story short of saying that even if we wanted to pay for a product like this that’s going to run in the anywhere from $10,000 to $15,000 range, we’re still working on that, the federal appropriation as is today, it would cripple it. So you would– a state would purchase maybe one of these devices, but then they wouldn’t be able to get their textbooks, their physical textbooks. So they won’t be able to get their Braille paper or other things like that because they’ve already spent the money on this. So what we started to do is work with the Department of Education and our federal government connections to essentially, increase or create a separate fund that is dedicated specifically to high-tech initiatives. Because even today, what’s happening is we’re seeing states choose not to purchase technology with the federal appropriation funds because they have to sacrifice too much in order to get that technology. So the way that anyone who’s listening here in the United States can help is go to your representative, your regional local representative and tell them the story of why a multi-line professional Braille display would have changed your life if you were a blind individual in education, or if you were working with kids today, how that could impact their education today.

DEVIN COLDEWEY: And it’s sad that there has to be such a balancing act for something that would be so plainly helpful to so many people.

GREG STILSON: Yeah. It’s a challenge. But I will say, at least the initial meetings that we’ve had with the folks at the Federal level have been really, really positive. I think this is a story you don’t really need to know much about our field to understand. So that’s been really positive.

DEVIN COLDEWEY: Good. Well, I hope that those talks go well. Speaking of talks, we only have about two minutes left here, and there’s a couple more things I want to ask you about. You mentioned that you’re working on, in addition to the new tech and the new software, you’re working on a new ebook standard for Braille books. How does how does that work? I’m really interested in this, like, this kind of file format stuff. I don’t know. It sounds a little dry, but honestly, I think it’s really cool.

GREG STILSON: So it’s another example of something that’s not changed in 40, 50 years. So the standard for Braille today is the Braille Ready Format or BRF. Or in some countries, it’s BRA, BRL. It’s basically a .txt file for Braille, essentially. It’s an ASCII text file. There’s no markup. There’s no– and what’s really funny is that our Braille transcribers today, they use tools that actually create markup. They’re saying this is a table. This is a heading. But then when they say file save as BRF, all that markup gets thrown away. And it’s just the ASCII text file, essentially. And the reason being is because it’s used to be embossed. It’s embossed on paper. So the first problem we had to look at is if you’re using an ebook reader, how can we take advantage of the standards like EPUB and Daisy today, and we’ve been really lucky to have a ton of interest from over 30 organizations and countries around the world, along with the Daisy consortium who’s helping us basically create this new EBRF standard which gives you the full integrity of formatted Braille while utilizing the markup that you’re given from things like EPUB and things like that. So now, if a student does want to go to chapter 5, they can do a command on their device and say go to chapter 5, and it’ll jump to that at that point.

DEVIN COLDEWEY: That’s awesome. Yeah, I love that we’re updating. I mean, it’s long, long overdue, like you said. We just have a couple of– just a second left, but I know that you’ve got, coming up this year in addition to this tech, we’ve got an SDK coming out. Is that right? Can you tell us a little about that?

GREG STILSON: Well, coming out as a strong way of putting it, but we’re currently working on it. APH and Humanware have never built an SDK, so we’re kind of building the airplane while we’re flying it here. But we are– our hope is this is going to create and generate so many new ways of interacting with tactile experiences, right? And APH is not so bold to think that we can come up with every single one of these. I know there’s brilliant people out there that have a ton of opportunities that they could think of to create new experiences. And so we are building a software development kit that will initially be released to what I’m calling a trusted partners circle of organizations. Folks who want to get involved can email DTD Dynamic Tactile Device, DTD@aph.org. And I see all of those emails. Let me know if you’re at all interested in developing for this. But the reality here is that we know that this is just the beginning. And in order to really have this technology reach its full potential, we need to unlock the full development community to start to come up with some really awesome ideas. I am most excited for what we can do with tactile gaming. That’s something that as a blind student myself, I did not get a ton of gaming opportunities. And so I think about our students today and how gamified learning is and how many of our blind students are not able to participate in that. So really excited about what’s to come on that.

DEVIN COLDEWEY: Absolutely. Yeah, a lot of things to be excited about. I can’t wait to see how it all comes together. But that is the end of our time. Thank you, Greg, for coming and talking to us about this and really looking forward to hearing where it goes next.

GREG STILSON: Oh, thanks for the opportunity, Devin, and thanks for everybody’s interest.

[MUSIC PLAYING]