Spotlight: APH – Monarch

[MUSIC PLAYING]

VOICEOVER: Spotlight, APH Monarch. Speakers, Greg Stilson, Head of Global Innovation for APH, and Jason Martin, Technical Innovations Product Manager for APH.

GREG STILSON: Hello, and thanks for joining us for the Sight Tech Global presentation on the Monarch, the evolving Monarch, and the goal being to get textbooks and curriculum materials faster for students who are blind or low vision. Just to quickly introduce ourselves, my name is Greg Stilson. I lead the Global Technology Innovation team here at the American Printing House for the Blind, and I think this is probably my maybe third year presenting here at SiteTech Global. And with me is a newcomer to SiteTech Global, Jason Martin, who joined our team last year here at the American Printing House for the Blind. Jason, what do you do?

JASON MARTIN: Yeah. I am a Technical Innovations Product Manager under your team, Greg, at the American Printing House for the Blind. And yes, this is my first year at Sight Tech Global, and I’m so happy to be here.

GREG STILSON: Awesome. And we are happy you’re here as well. So today we are going to do a little bit different. Since we’ve presented on the Monarch the last couple of years, we’re going to do a brief overview, but most of this is really going to be focused around our SDK, software development kit that we’re building and testing right now, showcasing a little bit of some of the apps that exist today and are going to be existing very shortly to really talk about the power of what the SDK does for software engineers, software developers that want to create really innovative tactile experiences. So our agenda today, we’re going to do a brief Monarch overview. We’re going to talk about what is eBraille and why that’s so powerful, the SDK, and then we’ll finish with some apps and demonstrations. So why are we doing all of this? Right? We, my team, I always tell the product managers that work for me, you got to focus on the problem and focus on the user, right? And so we always start with the problem. And what is the problem today? Right? And the problem is on the right side of this graph slide is a graphic of a student who is currently standing next to a algebra two textbook. Now that algebra two textbook is over. Right? It’s over 40 volumes and took over 12 months, almost 13 months to produce and cost over $60 ,000 the visual image here. And Jason step in because a blind guy doing a visual description isn’t really the best thing, but it’s a graphic of an or a graphic of these textbooks stacked up taller than she is. Is that correct?

JASON MARTIN: You got it, Greg. They are at least about six inches taller than she is.

GREG STILSON: She’s a high school student, and I think that a lot of this graphic is pretty reminiscent of what a lot of us blind folks dealt with and still are dealing with when it’s in regard to getting textbook textbooks on time and dealing with the sheer volume of these textbooks, no pun intended. So you know, when I was in school, those textbooks pretty much took up a room, right? We had a resource room that most of my textbooks were stored in. And beforehand, I would have to go to the teacher and say, all right, what pages are we dealing with today? And then I had to run down to the resource room, find the correct volume and bring it back and make sure that I also had the right tactile graphics with those specific pages. So students receive materials far later than their sighted counterparts. It is increasingly expensive and really long lead times for students to get access to their textbooks. Now, textbooks do come in chunks. Sometimes you’ll get four or five volumes at a time. But oftentimes, it’s after the teacher, classroom teacher, has actually gotten to those pages. Students have very limited access to tactile graphics. We when we were doing our research for the Monarch, we would hold focus groups with blind adults ranging from really short of young adults all the way to seniors. And the number one thing we ask them is, what’s your experience level with Braille and what’s your experience level with tactile graphics? The vast majority, especially of the adult users, said that their experience level with Braille was pretty well, pretty high. But their experience level with tactile graphics was limited to none. And that’s really short of what we’re seeing in this space is that access to tactile graphics is extremely limited. The average teacher that we talked to told us that they teach around 20-25 tactile graphics in a year, 25 tactile graphics in a year. A sighted person sees 25 tactile graphics pretty much when they open their eyes or 25 graphics when they open their eyes, right? So we’re dealing with a severe tactile famine for blind and low vision students. There is an extreme lack of spatial understanding and formatting for students with visual impairments. And that that comes from a few things. Number one, that students need to receive hard copy material to be able to understand spatial page layouts, spatial formatting, things like that. And then, as we said, you need to have tactile graphics under your fingertips to understand spatial concepts like geography, like maps, like routing, those type of things. Without a tactile image of those things, you are you’re not going to get a quote unquote visual sense of what it is you’re looking at. There are extremely complex workflows. In STEM and STEAM subjects, by nature, STEM subjects are very visual, right? So without those tactile renderings, it’s pretty challenging to get get an understanding of what the spatial concepts are that you’re looking at. And there’s really poor impromptu learning options. We know that our teachers of the visually impaired today are, you know, they all have their Mary Poppins bags and they’re able to to create impromptu graphics on the fly when the gen ed teacher gets to provide the TVI a graphic or information ahead of time in an accessible manner. And so these teachers of the visually impaired are left to try to keep learning happening on the fly, right? They’re doing impromptu learning exercises. They’re creating tactile graphics out of rubber bands and thumbtacks and puffy paint and anything that they have in those Mary Poppins bags that they can pull out on a whim and create these graphics so that learning continues on the fly. But we know, that’s today we can do, we can make this more efficient, right? We can give them additional tools to really expedite this process. So what is the Monarch? I just have a visual image of the Monarch here. And I’m going to quickly verbally walk through the layout, but the face of the device is taken up by the Monarch’s 10 line by 32 cell tactile display in the top left corner is a page up button. Bottom right corner is a page down button. Those are very strategically placed so that ergonomically, as your fingers naturally go to the top left of the display, you’ll find that page up button. And as your fingers reach the bottom, right-hand corner, as you’re concluding that last line of Braille, your little finger on your right hand is naturally going to hit that page down button. To the left and right center of the tactile display is what we call the D-pad or directional arrows. Those are used to move your cursor around, move focus. Or navigate a zoomed in tactile graphic for panning, for example. At the bottom left and bottom right of the tactile display is a zoom out and a zoom in button so that you can zoom in and out on graphics. Below the display is a Braille 8-dot entry keyboard with a space bar. And between dots 1 and 4 is an action button that’s used for multiple things, including a gesture that we call point and click. So that you can use touch to actually zoom in and out on graphics. On the left edge of the device is your USB-C charging cable connection or data connection, power button or sleep button, and a USB-A port. On the right side is a HDMI connection, volume up and down, and a headphone jack. And on the front face of the device, or I’m sorry, front edge of the device, is your traditional back home and recent applications button. The Monarch is about the size of a 15-inch laptop, gaming laptop, a little thicker. So let’s talk a little bit about the overview. So it is a revolutionary tactile display capable of rendering tactile graphics and Braille characters on the same surface. It’s developed by APH and Humanware as co-investors of this project. We worked, we partnered with the National Federal Government. We worked with the National Center for the Transformation of the Blind to ensure that we kept the voice of the blind person at the center of every decision we made. The tactile display uses cell technology developed by Dot Incorporated out of South Korea. So we use their tactile display in our device. And that’s really the innovation that allows us to put graphics and Braille on the same surface. We did add significant technology to their Braille tech. Including a touch screen, or a touch sensor over the top of the display. Along with some really powerful compute underneath. So what does Braille on Monarch look like? Well, like I said, it’s up to 10 lines of 32 cells of Braille. And I say up to because the user using the technology we have underneath has the ability to customize how many lines of Braille you want to see on the device. And we did that intentionally. Because we know that some Braille readers advance slower or faster than others. Traditionally, when you’re learning to read Braille, you read double line spacing. So oftentimes here at APH we’re the largest producer of books in the world. And we have to oftentimes emboss the same book twice. One is double space and one is single space. For the first time ever a blind user is able to customize how much space they have between the lines. So if they’re learning Braille, for example, they can learn that first in double line spacing and get to understanding uncontracted and contracted Braille. And as they move forward, they can reduce the amount of spacing between the lines just by tapping the zoom out button a few times and you can go anywhere from ten, eight, seven or six lines on the display at one time. The eBraille support is coming later this year. It’s going to be a lot faster. It supports traditional BRFs or BRL files, and it also supports formatted computer-translated print files. These are things like Microsoft Word files, PDF files, TXT files. So taking the print files that you would traditionally open on a computer, you can open those on Monarch, and they will automatically be translated into Braille. As we said, tactile graphics are able to be rendered on this device as well. Braille uses 3,840 equidistant pins to render tactile graphics. Like I said, we can use those to render both Braille and graphics on the same surface. So you are able to take graphics either by tactile artists or what I would call visual images, JPEGs or things like that, and put them on the device. Now, what we always tell people is you are going to get the best graphic experience by looking at tactile graphics graphics. And that’s created by tactile artists, right? We can take a Google image of, say, somebody’s face or something like that, right, and put it on here. But the amount of noise in that image is going to really render it tough to use as a tactile user, right? We do have some filters in the device to remove detail and to kind of put edge detection on it so that you can see outline drawings and things like that. But the reality is the experience. The experience is just going to be exponentially better when you’re using a tactile graphic created by a tactile artist. And that’s why we built in a direct API call or connection into the tactile graphics image library. So because this is a connected tablet, we have the ability to, within the Tactile Viewer app, basically instantly pull up graphics from the over 2,500 tactile graphics available in the Tactile Graphics Image Library. So if you want to do that, you can do that. If you want to see the image of a car or the outline of the state of Washington or something like that, if it’s in the TGIL, tactile graphics image library, you can search it and immediately pull it up on the Monarch. So like I said, tactile graphics will render the best on this device. And the other reason is because tactile graphics have Braille labels right there. So you can actually pull up a map of the United States, see the state’s labels, and be able to access them instantly. It does support PCS. It does support PDF, JPEGs, and PNG images. And we are working to support SVG as well. And then Jason is going to talk about our new upcoming Wing It app, which allows a non-tactile graphic artist to be able to generate tactile graphics instantly on the device as well. The other really cool thing and super innovative piece that we’ve partnered with the Desmos Group. Graphic Calculator to do is: we can instantly create tactile graphs using our key math graphing application on this device, which uses the Desmos API to create instant Cartesian graphs on this device. So for the first time ever, a blind student can graph an equation, recognize that maybe they messed up something or they want to change that minus two to a plus two, and have it instantly re-graphed to see exactly what happens there. So what are the foundational apps found on the device? So you have the Braille Editor. What you Braille is what you get. I always tell people it’s like a digital Braille writer. Tactile Viewer with TGIL integration, as we were talking about. This is the graphics viewing application that allows you to look at the graphs, zoom in and out, pan around the graphs, things like that, or graphics. The Word Processor is like a multi-line Microsoft Word. You can take . exe files and open them up. You can instantly see them in multi-line form. You can do editing. This is what students are going to do most of their homework on. As I said, the Desmos-powered graphing calculator application. And you can take those graphs that you produce, copy them to your clipboard, and you can actually paste them into that Microsoft Word file. So if you are doing some math homework and your teacher says graph this expression, you can graph it, copy it, and then paste it back into the Microsoft Word file. That Microsoft Word file, or Microsoft Word app, as we said. Does also support MathML. And the real exciting part here is that if a teacher, professor creates a math worksheet, and they use the equation editor in Microsoft Word to create some math, that math will instantly be translated into your preferred math Braille code, whether you use Nemeth or UEB math. So you basically open up that . docx file that has math in it; it will instantly translate that math into Nemeth or UEB. So you can print that math in your device, and you can answer any worksheet or questions, and then immediately save it back as a . docx file, give it to the teacher, and when they open it in Microsoft Word, they’re going to see printed math from your Nemeth or UEB math that you wrote. It obviously has a file manager, and then the Victor Reader book reader application is the app that’s basically going to be used to read any textbook, to read literature books, things like that. We do have direct integration with Bookshare. org and NFB Newsline as well, and this is what’s going to also support the eBraille format coming. It does have an internet browser on the device, and then, as I said, we are working with screen readers on the device, or screen readers in partnership to support this device as one of the first multi-line braille terminals as well. And this is really a challenge. Right. But a lot of folks are saying, why doesn’t that support exist already? And it’s sort of a chicken or the egg thing, right? There has not been multi-line tactile displays to enable the screen readers to output both multi-line Braille and tactile graphic representations yet. And so that’s what we’re doing right now. So we are in active conversations with all of the screen reader providers. Quickly, what is eBraille? We’ve talked a lot about this. eBraille is a. A standard that has been created across the world with APH, the Daisy Consortium, and over 40 international organizations. The goal is to create a really powerful, useful digital Braille format that keeps all the integrity of Braille formatting while bringing you the power and navigation support of a modern electronic format like EPUB. So the advantages of eBraille. Today, if I were to take a BRF file, and you can do that right now on the Monarch. You can take a BRF file. The problem is BRFs are hard-coded to specific dimensions of a piece of paper, right? So most eBraille files, or I’m sorry, most BRFs are coded to either 11 1⁄2 by 11 sheets of paper or 8 1⁄2 by 11 sheets of paper or letter size. When you display them on a display that’s 32 characters wide, if you’re displaying a 40-character BRF, you’re going to get this. You’re going to get this strange long-line, short-line scenario where it’s trying to wrap the remaining 8 characters onto the next line. And it makes sort of a disruptive formatting at that point and kind of messes up your braille formatting. So we wanted to do better. And so we worked with the Daisy Consortium and, like I said, the eBraille committee to basically create a standard that is electronic, that is sort of device agnostic. It doesn’t need to know. It doesn’t need to know what device you’re using. And it uses markup, right? So you’re able to use all of the tagging that we do in EPUB and Daisy and all that kind of stuff and actually tag braille, real formatted braille, right? So if you have a heading, let’s say you have a cell 5 heading or things like that, that’s going to be tagged as an H2 or an H1 or whatever you decide to do. You can tag tables for navigation. It uses Unicode instead of ASCII. So that it allows us to share internationally, contains graphics within the actual file, right? Today, when you’re producing a textbook, what ends up happening is all the Braille portions of that textbook get produced first, and then you have the tactile graphics put in second, and they end up having to get folded in later when the book is being produced. Here, there are going to be direct links into those graphics within the file. So you’ll see a link referencing a graphic. You’ll be able to, within the Monarch, use that point-and-click gesture or press the Enter key on that link, and it’ll immediately open up that graphic from within the book. It’s going to benefit all multi-line and single-line displays, right? As I said, it’s going to be device-agnostic, and it’s also going to benefit embossers as well. It’s going to open new possibilities for digital delivery of content, whether it’s textbooks or curriculum material, so that we don’t have to wait until the entire book is done. For sections of that book to be immediately digitally delivered. Long term, our goal is to support eBraille in transcription programs, and that is coming in early 2025, right? We’ve already partnered with Duxbury as well. The folks at Duxbury have committed to providing eBraille support in 2025. And like I said, I think it’s going to be early 2025 so that you can create your eBraille files from within Duxbury just as easy as you could create a BRF from within Duxbury. We’re also going to be adding this support into Braille Blaster as well. And we’re going to be, as eBraille support continues, you’re going to start to see these born-e Braille files. But in the meantime, APH is working on an eBraille converter. And the goal. Of this converter, is to basically take all the hundreds and thousands of millions of BrFs that exist today and give you a path to turning those BrFs into E-Braille files because we can recognize formatting just from the spatial alignment that has been created in these BrFs. So, we’re going to be open sourcing a tool that is going to give you as the user, you can run either a single or you can batch convert a number of Brf files into E-Braille files so that they can be used immediately on tools like the Monarch or other refreshable Braille devices. Now, we’re going to talk a little bit about the software development kit. One of the earliest initiatives that we did when we started this Monarch project was, we with it being one of the first tools out there ever to date to do what it can do. We said early on, ‘we don’t know what we don’t know right.’ We know many of the use cases where this is going to be valuable but we don’t know all of them and we wanted to make sure we created a path where people way smarter than us can create experiences, tactile experiences and Braille experiences. On this device, that are just going to change the game right? Um, so what we did with our partner Humanware is we we started down the path of building software development kits. So any software developer who knows how to develop for Android can can create a a tactile experience for for the Monarch and right now, this uh, this SDK software development kit is in beta and it’s being tested currently by APH. So, Humanware built the SDK originally and now APH is using it to build some pretty cool uh apps and experiences that a couple of them are are available on the Monarch today, so that’s where we’re going to be going with this one today. And there’s going to be one coming relatively shortly that we’re going to talk about. We’re going to be putting out a call for proposals in 2025, to basically give app developers the opportunity to suggest or or promote their ideas to gain access to the software development kit as well. We’re looking for partners not just software developers, uh, to build these apps. We want to elevate the work that you do and be able to put it on every single Monarch that’s going to be out there. So what I’m going to do now is I’m going to put up our con, our contact information, uh, on the screen here so I’ll put Jason and my contact information. gstilson at aph . org and jmartin at aph . Org, feel free to reach out to us with any questions or app ideas that you may have if you are thinking about what you could do as an app developer partner with us; um, feel free to reach out to us. But what I’m going to do now is I I’ve talked at you for a while, but now I get to turn it over to Jason to talk about uh some of the really cool things that we’re doing uh with the software development kit and really to showcase what this SDK is capable of because the reality is that you know we’ve we’ve built a few experiences but but we are barely scratching the surface and Jason gets he gets the fun job of working with all of our software for engineers to build these experiences. So I’m gonna stop share here and allow Jason to share his screen directly from his Monarch.

JASON MARTIN: Great, thanks, Greg. I’m going to find my Monarch screen and here we are. And I’m going to ensure that I’m sharing sound too because that always gets a little goofy. My narrator on my Monarch is British because I like to know my Monarch apart from the other ones sometimes in the field, but what’s the voice of the default Monarch, Greg? Do you know? Say that again? The default voice on the Monarch, the lady’s voice that comes up. Do you know her name?

GREG STILSON: I think it’s Lily, but I’m not sure.

JASON MARTIN: Anyway, so you will hear a British accent with my Monarch, but by default it is Lily or another female voice. Anyway, so what I’m pulling up on my Monarch and what you see on the main screen is the Monarch connected to a visual display. So while the user is seeing braille, and if I put it into braille view, we could show that and I will later, but what you’re seeing on the screen is a visual representation of the system of the Monarch. What I’d like to show first, and the first application that we developed in-house at APH, and that is the Startup Wizard. So for every Monarch, that releases in the Wilder, there’s a Startup Tutorial that begins, and we also included it as an application inside the system, so that if a new student gets the Monarch, or you have a student, maybe who struggles learning the Monarch, that you can go through this again, and again, and again, to really get the feel of the system. So I’m going to enter into Startup Tutorial, and I’m minimizing your face accent, Greg, for the time being so we can get just the maximum amount of screen here. I know we we love but so there’s a couple of things to note, um, and I love talking about that just the look of this main screen and it is something a bit different with the Monarch device traditionally, um, I would say really before the Monarch any startup tutorial or startup wizard in Braille has been one line and it hasn’t included tactile graphics. So in essence every startup application in Braille has been in text and in some ways we wanted to change that game. So on the screen you see in tactile font the words ‘Monarch Startup’ -it’s in print, it’s big, it’s bold, and underneath in Braille is the words ‘Monarch Startup’ to the right-hand side of the screen. You also See a power button that’s a tactile graphic underneath it’s labeled ‘power’. What’s unique about this, monarch startup font, and internally we jokingly call it the Godzilla font. And is that in braille, it’s very difficult to show emphasis, to show impact, other than bold, underline, and italics which are really the same code so you can’t do, in essence, comic book font or impactful things in braille. So we wanted to take a curious approach and provide a title screen the monarch. It enables us to do this really in real time with tactile graphics, in braille text, and interactive braille text. So it’s giving me the option To press Enter, I do welcome to the Monarch startup.

VOICEOVER: This tutorial will teach you the basics of using your Monarch. Press Enter to continue or Space with E to exit.

JASON MARTIN: It vocalizes; it reads in Braille. This section is also translatable to Spanish and French, so depending on which language you pick, it will translate automatically, and you can also use it to translate to other languages as well. So if you want to do it automatically, find and press each Pan key above, below, left, right, or D-pad; the Pan keys move the page up and down, and on this display, this is an example of including tactile graphic representation in um in the instructions.

VOICEOVER: So, we have images of the pan up and pan down key, and when you push those buttons since some screens have more content than can fit on the display, we use the symbol at the bottom right to let you know there is more Braille. Check out the symbol, then press enter to continue.

JASON MARTIN: When you push those buttons, it’s the system vibrates and there’s this really happy sound uh, to just do positive reinforcement. Also, for our um, for screens that extend longer, and in English they generally stay on the same pane, but if we were in Spanish or French, this naturally gets a little bit longer, so there is an arrow indicator down here at the bottom to tell us

VOICEOVER: hey there’s more text on this screen when a screen has more braille to read you scroll down using the pan down key when a screen has more braille to read you scroll down using the pan down key if you press pan down on the right side of the display the text will continue to move down until you reach the bottom if you press the pan up key the text will move in reverse until you reach the top try it out then press enter to continue and i press pan down and then enter ticket press each d pad key on the left or right side of the display press enter to continue so again this is orienting and greg did this with you verbally through the device

JASON MARTIN: But this is showing tactically, oh this is the feel of the buttons now go ahead and press those and again we get that positive reinforcement coming into play as in theized thing now for practice we glasses that something needs to be powered or not and once they are accepted, they don’t fit port into place um it’s a little bit gamified; the monarch fine compress each Braille key individually once each key has been pressed use enter to continue. One feature that I will highlight while we’re looking at this of the Monarch is you can see on the visual display where my fingers are actually touching the device, what’s unique about this. Is this was a development tool that was really designed to test the sensor where my fingers are interacting with the Braille, but as a teacher of the visually impaired, we also realized: wow, we can tell where a student is somewhat struggling with a Braille character or the impact for teaching Braille at a distance is very significant for the ability to monitor this at afar or even in the classroom to see well, is my student actually reading or are they just sitting there with their hands on the same spot? Here at the bottom is a is a very creative diagram of the Braille keyboard so instead of just playing whack-a-mole where we don’t know which key is missing that we’re pressing if we press a key we get that lovely sound reinforcement but also we’re able to update the image in real time on the screen so it’s listing the Braille keys in real time and tactilely has as I list them you, move through the menu using the d-pad find caterpillar and press enter I’m going to to slowly not not finish the entire startup tutorial but with this I do want to show you one instance of moving through first-letter navigation and so it’s asking for caterpillar, All right, so I hit caterpillar and I’ll say I’ve used the startup application with students I’ve used it with adults and I honestly say the adults are happier to see this image.

VOICEOVER: Check out this caterpillar, press enter to continue.

JASON MARTIN: Then, most of the students… It’s just a really creative way to start introducing the capabilities of this device and getting students familiar with it further, the device is my favorite section, and it is teaching the point-to-click gesture, and we call it the field of butterflies. We’re a butterfly appears on the screen, the user’s asked to find it to point and click on it. And then… Then it moves around the screen. And eventually, once they’ve done that enough times, the whole screen populates and it plays a happy tune. But this is really just orienting users to the device in a fun way. Greg, you got anything to add on startup?

GREG STILSON: No, I think what’s exciting about this today is that at the very beginning of this device, right, when you first turn it on, right, you’re exposed to tactile graphics right away. Right, which is very different than what students are experiencing today with regard to physical paper copies, right? As I said, the average amount of tactile graphics that teachers teach in a year is 25, right, 15 to 25. So this is really exciting that you’re able to immediately see tactile graphics. But you mentioned the point and click gesture, right? And this is a gesture that we knew was going to be a challenge because it’s not a gesture that blind students or adults have. It’s a gesture that we’ve used before. Sighted people use it all the time when you use a mouse on a computer. You point and click on things all the time. So, we took a page out of Microsoft’s book. And Microsoft, when they first developed Windows, they said, well, people don’t know how to use a mouse. So, we have to give them a tool to teach them how to use a mouse, right? And I kind of challenged Jason a little bit with this because we recognized that there had to be a way to teach a blind student in a, in an interactive way, how to point and click. And that’s where Jason had the brilliant idea to gamify it.

JASON MARTIN: So with all applications, we wanted to continue to reinforce this past the initial startup application, because going through a wizard, maybe the first, second time is kind of fun. But on the 30th time going through the wizard, it just loses its luster a little bit. So we wanted to find a way to incorporate this learning this new skill of point to click in a fun way, and we took a note from Microsoft. If you recall, the Windows 95 game of Solitaire that we thought Solitaire was just playing cards. But what Microsoft was so brilliantly doing was teaching us all how to point and click and drag and drop. And so in that vein, we’ve created my manual apps, Monarch Chess, Monarch Chess. And Chess has so many inherent benefits and playing the game as a chess nerd, myself, a former chess club captain. There’s so many great things about chess, but really it was to teach this this new gesture that’s not inherent to our students. So I’ve already got a game playing. Greg is already in the middle of it. There you go back out. Quick game because I love starting with the main time. Confirm this.

VOICEOVER: Main menu. New. Here we go. Monarch chess.

JASON MARTIN: So in the background you hear this very soft just park scene and individuals are setting up a game of chess. They’re rolling out pieces. So it’s just this pleasant aura of of what we wanted this this game to be. And again, you see my favorite font that says Monarch Chess in big, bold print at the top, and then it’s written in Braille underneath. And again, knowing that this localizes. So if you’re using this in Spanish or French. Monarch chess is going to stay the same. But underneath that Braille is going to translate into your native language on the left and right hand side of the screen, you see a white and black night representation in tactile graphic forms. And those are the physical chess pieces that what it what it would look like to use a physical chess board and underneath all of our tactile graphics. We labeled them again, black night, white night, or W night B night. So getting started through this game, Main menu. New game. One thing on the splash screen that that I love this, this video game title screen, if you will, conversations with our developers who are blind really were questioning, what’s the point in this? Like, why do we do this? Why wouldn’t we just enter it? Hit enter on Monarch chess and start into the game. And what we really wanted to do is recreate the feelings that average users get when they access typical software. If I access a chess app, if I access any type of multimedia experience, the standard is that there’s a title screen. So we wanted to replicate that as much as possible. So in chess, we realized that a lot of people don’t know how to play chess, and they definitely don’t know how to play our version of chess. So included tutorial, tutorial. We’ve sectioned out the tutorial. You could go through the full tutorial, which very much resembles the startup guide that you’ve seen. But. This is a lengthy tutorial, which goes through how to play the basic game of chess and then how our user interface works on the Monarch.

VOICEOVER: And so I’d like to just bring chessboard basics, how Monarch chess when chess is played on a classic board, it consists of 64 squares.

JASON MARTIN: The color of the squares alternate between white and black, how this differs just slightly from the Monarch startup. And I really like to show the graphical representation of what we’d like to call a classic chess board. This shows in the tutorial what a classic chess board feels like, what the quadrants of the chess board are. And it really is getting this idea that you can use tactile graphics in tactile graphic representation to teach physical games as well, like, and to teach concepts of how this game is played. So we show this image, and then the chess ball used to play on the Monarch does not have alternating colors.

VOICEOVER: It can be seen on the next slide.

JASON MARTIN: Then we show our version of the Monarch chess board, which notably chess boards alternate square colors. It’s black and white, black and white or red and black. But these it’s just a full grid that is empty.

VOICEOVER: So if you were to continue through this whole tutorial and you’re not pieces on the board, you will need to be familiar with the three types of tactile patterns we use: selection, movement, and placeholder. We will.

JASON MARTIN: But he stopped. Fantastic. So, we will show the user what our interface is, what those movements are, and how to interact. However, it is a rather lengthy tutorial because we are aware that many of our users haven’t played chess before.

VOICEOVER: Tutorial chess board new game.

JASON MARTIN: So, entering into a new game.

VOICEOVER: Choose opponent and play chess against CPU.

JASON MARTIN: The CPU just as a bit of information. His name is Kyle. We like to know that. And Kyle is very, very easy. And I’m not nearly as smart as Kyle. So I’m going to put it on the board. I’m very easy.

VOICEOVER: Monarch Chess. White Rook A1.

JASON MARTIN: Fantastic. So, in this view of monarch chess and I do have my team’s bar here at the top, it’s obscuring a little bit. But on the board, instead of piece representation and we do have a tactile graphic that shows what physical chess pieces look like. Naturally, we couldn’t fit all of the physical representation on the board and make it make sense. So, we went the route. First letter navigation or first letter naming of the pieces, should I say. So P for pawn right here in the center, B for Bishop, K for King, Queen or Q for Queen and then N for Knight and R for Rook. So it’s fairly easy if you know the piece names to understand what these pieces are and where they move, even for individuals who aren’t Braille readers. If you’re aware of where these pieces naturally set on the board, it’s pretty easy to associate with it. It’s just a different chess board. So I can navigate through the system in quite a few different ways. I could use first letter navigation and like P and cycle through all of my pawns. I could use my D pad and go White Pawn A2 move up or left.

VOICEOVER: White Pawn, White Pawn, White Pawn D2.

JASON MARTIN: And I could hit keyboard commands to interact with it. But what I really want to use this for is I want to put my finger on it and double tap.

VOICEOVER: White Pawn D2.

JASON MARTIN: And it makes this what I love about the Monarch is this interactivity. This the swish sound that you hear, the updating of the chess cells and tactilely you see this highlighted grid appear. So I know the spaces that my pawn can move now tactilely. And there’s a what I like to call the football shape where the piece was, this is where my piece was, this is where it’s moving. And without taking my fingers off the board, I can bring it up and say I want to move my pawn here. Point and click empty D4 at the bottom left and right hand side of the screen there is a pawn icon that’s actually a physical tactile icon it represents visually what it would look like and it alternates on left and right side of the board to show who’s playing the game so it’s just kind of a nice flourish but on the left and right side. And I’m not sure if I can get him to take my pawn, but if I were to take a piece or he were to take my piece or the AI were to take my piece you would see this populated on the left and right side of the board so it’s effectively the graveyard, if you will, it’s a morbid term, but that’s the name. Just as an kind of just to show you the capabilities of this chess board the pawn is a very small piece, it only moves two spaces, but if I click on say the

VOICEOVER: Bishop White, bishop see one.

JASON MARTIN: The bishop has a significantly bigger moving pattern, and well maybe I don’t want to do that. I’m going to hit space and sit it down.

VOICEOVER: White bishop c1.

JASON MARTIN: The knight is a considerably more complex piece to move.

VOICEOVER: White knight b1.

JASON MARTIN: And it moves in very similar fashion. So the entire game of chess is playable. You can play it against a friend in the room, so local multiplayer. You can play across the board or you can alternate the monarch side by side to play. Chess is not finished. I’m, I wouldn’t let it be done until there was a version of chess that could interact with an iOS device or an online existing platform of chess. So our students can play chess with their peers, and their peers don’t necessarily have to be Braille readers to play chess with our monarch users. So we really hope this next extension of chess is really going to excel the game right now, chess is out with the monarch and fully playable. The online interactivity and the peer display support are definitely coming next year. Greg, anything else about chess that’s that you you can think of that’s just incredible! I love I hope tons of schools start playing and start having chess clubs. I know Washington and Texas have promised me that they’re starting monarch chess clubs.

GREG STILSON: Now that’s awesome, now you know I think in this case what what this showcases is is the capability of a chessboard of monarch there’s a couple things that we built into the SDK and I think the the tutorial best shows it is the ability to do, you saw the Braille characters but they were verbally immediately translated into print uh or into to to standard um standard text right so we had the Braille characters in the tutorial but it also allows you to mix Braille and tactile images in the same fashion that we showed the UI that would be available in the in in monarch chess and be able to represent patterns and represent different Symbols and graphics alongside the Braille that’s on the screen, and I think that’s really exciting because we’ve never had that ability right when when we read a tutorial or if we listen to a tutorial it’s always only text, right? You never get any graphical representation, and I got to give Jason a ton of credit because when we first started building the startup tutorial or the chess tutorial, right, it was traditional text, like it was the way that we always received information, and Jason basically said, ‘like, we can do better than this. We can make this much more interactive; we can turn on images and bring graphics To the user’s fingertips, right away, so the SDK gives you access to the entire tactile array to do with it what you what you want to do. You can have um, you can have you know print characters translated into braille; you can have braille characters translated into print; and you can on the same display designate sections for graphics, uh, where you can create your own graphics. And that’s really what we’ve done here with the chessboard: is we’ve created a graphic of a chessboard and put pieces, uh, in inside of those those boxes right? The one thing that you’ll notice in this as well, the other sort of interesting piece is because we Are using Braille characters, how do we create a graphic of a chessboard so you’ll see an extra dot below the ‘P’ of pawn as an example and that’s how you you’re able to sort of differentiate the two, I think um you know you also gain gain access to all of the sensors on the device, the touch sensor, you’ll gain access to the the haptic motor, all the visual output, text-to-speech, Braille translation capability with the software development kit. So extremely, extremely powerful. Like I said, we don’t know what we don’t know. So we’re excited for you all to come to us with some incredible ideas. And speaking of incredible ideas, I think that the last thing that we’re going to demonstrate quickly is one of the biggest requests that we’ve had from teachers of the visually impaired. And we talked about that Mary Poppins bag that they carry around, right, to create these impromptu experiences or graphics when the gen ed teacher did not provide them the information ahead of time. And so I sort of threw Jason the challenge of creating this impromptu drawing app. And the target use case is really a teacher of the visually impaired instantly drawing a graphic for the blind student on the fly and having that graphic on the fly. We’re going to expand upon that moving forward. But for now, that’s the initial challenge. And that gets us so much further than where we are today with wiki sticks and rubber bands and puffy paint and things like that. So I’m going to turn it back over to Jason to talk about the Wing It app and where we’re going with this.

JASON MARTIN: Yeah, Greg, thank you. I actually pulled it up while you were giving that debrief on chess, which is awesome. And I just got to add, like, you know, it’s the interactivity of being able to point and click in Braille. There’s something about that, Greg, that, like, really has enhanced all of these interactions that we’re seeing just, I think, without that ability that we wouldn’t see these applications being utilized or the SDK being utilized the way it has been. I think it’s just an exceptional part about this. So I’ve pulled up the Wing It app. But visually, I’ve got to explain that on the Monarch, right now you’re seeing a demo version or a definitely a beta version of this application. And if I were to do this with the visual display running, you wouldn’t see anything coming from it. However, when I’m on the Monarch at any point in time, we can open up what’s called visual Braille. And I’m going to do that real quick.

VOICEOVER: And options menu. Miscellaneous setting. Miscellaneous setting. Check.

JASON MARTIN: So now on the visual display, what we’re seeing is a one-to-one representation of the Braille characters on the screen.

VOICEOVER: Options menu. Miscellaneous settings.

GREG STILSON: This feature is super powerful for teachers visually impaired because you no longer have to be always looking over your kid’s shoulder at the tactile display. If this is connected to a monitor, you can be following along with what they’re doing in real-time without having to awkwardly look over their shoulder.

JASON MARTIN: So with this tactile graphics drawing tool or the Wing it application, what you can do is in real-time, and I’m just going to draw a rectangle in real-time. Say I’m drawing a classroom layout or a campus layout. It’s really one of the use cases we see in schools is that students have to know where their classes are, where the lunchroom is. And if we needed to do this traditionally, you would have to draw this type of graphic. But I have an iOS application on my phone and I’m just going to let go. And then, I could draw a classroom and this appears tactically or the campus map. So, I’ve been able to draw college campuses in quite literally seconds to give a representation of this. Now, not only can you draw simple shapes and things like that, but we need to have patterns and lines and other ways of distinguishing information.

GREG STILSON: So just to clarify what Jason is doing right now is he’s using an iPad or his iPhone, Bluetooth connected to the Monarch tactile display. So, the teacher, the visually impaired would have an iOS device. Our goal is to bring this to Android down the road as well, but for now, have a iOS device. There was literally no virtual, or not virtually no pairing process right there, there’s some stuff happening under the hood. But Jason literally just pressed the connect button on the Monarch and it connected to the wing. It’s an app that’s running on his iOS device. And what he’s doing now is just drawing on the glass screen of his iPhone.

JASON MARTIN: And that’s a that’s a great point, Greg, because in you know, if you’re in conference centers or in our case, if you’re in classrooms, sometimes accessing Wi-Fi can be just extremely difficult. You have to jump through a million hoops sometimes and this just directly connects. And that’s what we knew. If a teacher is working in a classroom, they don’t need the wireless access as a hurdle. They just need. To be able to access it with the student. Now, what I love about the Wing It application is so now what I’ve drawn on the screen is just kind of a representation of, say, a city street or a campus map with lines indicating streets. But we all know maps get bigger than this. It doesn’t maintain the same ratio. So I can pan the screen. I wanted to and add more information to it. And ultimately, when the Wing It application is finished, students on the Monarch will be able to pan through these diagrams, save these diagrams and ideally label these diagrams. So if we need to know that this is the math classroom, we can see this drawing, look at it, click and label it in Braille. Now, there are numerous fill patterns. The developer on this is went just in wonderful ways, just very in depth to be able to show various fill patterns and shapes because we need to be able to distinguish what. Textures one texture from the other. So as I pull up a star. Yes, this is a great representation. I’m just going to put a star on here with a blocky pattern in the center, and I’m just drawing a few stars to give you this idea that. The star has a block pattern. If I wanted to fill it with some other type of fill pattern, I can change that element and then add on, say, another star, which brings it diagonals. Into the fill space. So there is a tactile distinction between these two shapes and in numerous ways you can use this application. I’ve used it with what we would expect is inside the classroom. If I hit clear. I can just quickly clear out the diagram, but say there is a Cartesian graph that our students did not get access to early on. We could. Then come in and draw that Cartesian graph in real time to a pretty fair approximation. Say we’re studying slope and intercept, so that is just a real-time representation of something. Maybe it’s a concept I’ve spoken with developers about networking, just computer networking. This is an easy way to quickly draw out a networking map and save it. It’s a fantastic application, and we’ve really only seen. Just a taste of what is going to come from this. I can’t wait. I’ve seen teachers draw Japanese kanji guide dogs. I play Pictionary with blind individuals, and it’s been very hilarious, so there’s so many uses and applications for this that we really are only scratching the surface, and it will be coming out at the beginning of next year.

GREG STILSON: Fantastic. Well, we want to thank you all for spending your time with us. These are just a few of the apps that are on the Monarch. As I mentioned, these are not the foundational apps. The foundational apps are the ones that I already had went through, right? The tactile viewer, the graphing calculator, the word processor, all of those things, right? These are additional apps that have been produced strictly because of the implementation of the software development kit. And then, like I said, this is something that we’re going to be looking for partners with. Moving forward. So we hope that you’ve enjoyed this time with us. We hope this has been informative. And please don’t hesitate to contact us if you do have future questions or ideas, thoughts on how to partner, any of that kind of stuff. And we’ll look forward to speaking to you at the next Sight Tech Global. Thanks so much.

[MUSIC PLAYING]