Inventing the "screenreader" for VR: Owlchemy Labs' Cosmonious High

For developers of virtual reality games, there's every reason to experiment with accessibility from the start, which is what the Owlchemy Labs team did with Cosmonious High, the 2022 release of a fun, first-person game situated in a inter-galactic high school that one reviewer said "has all the charm and cheek of a good Nickelodeon kids show." And it reveals some of the earliest approaches to accessibility in VR.

[MUSIC PLAYING]

BRIAN: Inventing the screen reader for VR. Owlchemy Labs’ Cosmonious High. Moderator, James Rath. Speakers, Peter Galbraith and Jazmin Cano.

JAMES RATH: Thanks, Brian. James here. So I’m super excited to share this conversation with Jazmin and Peter from Owlchemy Labs. This is one of the things I’ve been interested in for a while, is how do us from the blind community enjoy virtual reality? So if you don’t mind introducing yourselves. Jazmin, we’ll start with you.

JAZMIN CANO: Yeah, thank you. So I’m Jazmin Cano. My pronouns are she/her. I’m the Accessibility Product Manager at Owlchemy Labs. And my job is to ensure that accessibility features are researched, tested and prioritized. And I’m very excited to be here.

PETER GALBRAITH: Hi, and I’m Peter Galbraith. I’m the accessibility engineer at Owlchemy Labs. I’ve been working on designing and building a lot of features and systems for accessibility in our games. And my pronouns are he, him.

JAMES RATH: So we’re here today to talk a little bit and show some of the experimental options that you guys have been including with your game, Cosmonious High. It’s a really well acclaimed VR title. And it has been creating reputation with being accessible with other features. But now you’re exploring opportunities in vision accessibility. Could you talk a little bit about the game itself, but also how you came down this route of wanting to make your game more accessible for low vision users on VR?

PETER GALBRAITH: Absolutely. So Cosmonious High is a game about being a new kid in an alien high school. It’s there to be inclusive. We’ve created this wonderful friendly environment. And we’ve been doing some accessibility work in Cosmonious High. And our previous titles, Job Simulator and Vacation Simulator as well. And we believe that accessibility is very important to the future of the industry, in both video games and also in virtual reality. And technology in general.

JAZMIN CANO: And to add to that, after each game, Owlchemy Labs has been taking a big swing at unsolved areas of accessibility and XR. So previously, the team looked at subtitles. And this time we ask the question, could a user with low or no vision play our VR games? So we don’t know if our efforts will be successful yet. We feel it’s important to try. And while our solution is not comprehensive for all users with vision-related disabilities, we feel our vision accessibility update will make huge strides towards making XR more inclusive.

JAMES RATH: That’s awesome. One of my favorite things about this whole thing is a lot of folks who are getting into accessibility are oftentimes learning as they go. Listening to the community or getting those folks who need these features involved with those projects. Can you share with me some of the challenges that your team faced when working on this initiative?

JAZMIN CANO: Totally. So one of the bigger challenges was play testing. At the start, we wondered how can we reach our target audience for play testing? And it took a lot of searching. And we finally found a third party called VR Oxygen. We work with them to find play testers that match the profile of the people we’re looking for. So starting that relationship was awesome because finding organizations and companies who can help test for accessibility, specifically in VR, that was a huge challenge. Additionally, it’s taken months to actually find people with the equipment that’s required and who can work during our office hours as well. That was another challenge. But thankfully, we’ve had a number of play testers that resulted in lots of learnings for our team. And discoveries we can share with others in XR working on this.

PETER GALBRAITH: Yeah. There were a lot of interesting challenges there. Remote play testing provided its own set of technological hurdles that we had to overcome. But I think through clever use of technologies like Zoom and being able to screen share and do things like that, we were able to effectively mitigate the challenges that arise from that. Beyond just the testing sides of things though, we also had a lot of challenges in the design and implementation process. So one of the big things here that was a challenge that was mentioned earlier is that Cosmonious High is an already released title. So we’re working on a finished product. Which means there is a lot of stuff that has been locked in and can’t really change too much. It’s systems built on top of systems. And so it becomes challenging at times to add a new feature without breaking or changing the core experience of the game. It’s been a bit of a challenge, but we’ve been working on it. And I think we’ve been doing a pretty good job of building this into our systems. And hopefully in a way that it’s something that we can bring forward into future titles. Additionally, there’s a lot of design challenges around the technology itself. So a big thing we have here is our 3D screen reader sort of technology. It’s meant to allow us to get any object in the game of importance. And provide auditory feedback on what that item is. And give the player context about the world around them. It’s a little more challenging than your traditional screen reader as it needs to– in a traditional screen reader, you’ve got a flat page layout and the screen reader can go through the different elements. In a specialized environment such as VR though, where everything exists in this virtual space, getting the information you want is challenging. So we use indicators like hovering your palm over an object or pointing with your fingers at an object in the distance to help establish intent of what you are trying to get information about. And I can show a quick clip of that if we want. Let’s do that.

JAMES RATH: Yeah, that’d be great. [VIDEO PLAYBACK] – Hovering over new student ID card number five. A student ID with a purple edge for the new kid. Hovering over new student ID card number four. Hovering over new student– hovering over new student ID card number three. [MUSIC PLAYING] Aiming at registration field. A glowing platform. The same file you are loading is determined by which student ID you place here. Aiming at registration field. Hovering over subtitle drawer. Grabbed subtitle drawer. A pullout drover containing a pop up tool for selecting subtitle options. – Pick an ID. Any will– [END PLAYBACK]

PETER GALBRAITH: So what you can see in this video is object highlighting and descriptions based on the object that is being pointed at. So the pickup-able objects have a light blue highlight around them. Other key interactive tools have this green border that helps you understand that there is something to do here. So if you’re a person with low vision, there’s a good high contrast outline. And something that doesn’t show up in these videos is that there are also haptic pulses in the controller to help you feel when you are on or off of an object. And know that you can trigger the request for more information from our screen reader-type technology.

JAMES RATH: Do you mind if we talk a little bit more about the controller of VR? So just for those who haven’t been maybe– for those who haven’t experienced VR. Essentially whether it’s an application or a game. You’re wearing a headset that can track your motion orientation. And you can move at a 360-degree angle. And with those, you tend to get two different wireless or wired-motion controllers with analog sticks and a few buttons for input. But a lot of the input in these types of applications is with your full body motion. Can you talk a little bit about what opportunities do you have to describe what gestures need to be done for maybe a player with low vision or blindness?

PETER GALBRAITH: Yeah, absolutely. So one of the interesting challenges is that many of our tutorials had been very heavily reliant on visuals in Cosmonious High. So teaching the players what button to press on the controller was challenging in an environment where our users may not be able to see the graphics or may not at least be able to see the graphics clearly. So what we’ve done is we’ve developed some language to help us describe these actions spatially to our players. And we do that based on their own physical body in a lot of ways. So when they’re holding a controller, there are face buttons on the controller. We can tell the user which face button to press by saying, OK, there’s two face buttons on the controller. Press the one that is closest to your wrist. And by having that sort of a mapping, by having a sort of a mapping, we’re able to indicate what the player needs to do. Additionally, one of the things that we wanted for this was we wanted players to be able to interact with this game and use these features without sighted assistance. So for turning on the low vision mode that we’ve created, it’s as simple as telling someone hold the controller to your ear and then double tap the controller face button that is furthest away from your wrist. Or whatever the appropriate button is on the variety of controllers that we support. We’ve had to create a language around all of those. But by doing this and having these interactions based on relative to the player’s own body, they don’t have to see as much of the environment early on to begin this sort of journey with our accessibility features. And from there, we have a lot of different ways of interacting that we inform the player of telling them either to hold their palm at something and then press the button or aim their fingertips and press the button, based on the context. And doing that seems to help a lot.

JAZMIN CANO: I want to add to that.

JAMES RATH: Yeah, please.

JAZMIN CANO: Yes. So I want to add to that. Early on in our initial designs, it was co-designed with someone who has blindness and something we use in our game for people to read. They can do a hover where their palms facing the object. Or they can point, like Peter said. And I thought it was really cool to hear that this person said they wanted it to almost be like you’re using the force. So they know what’s around you in this world. And so those are additional ways to describe it as well, is just hover your hand over something or point at something. So we’re assuming folks would then put their palm out and point their fingers out.

JAMES RATH: And with that, haptics obviously play a big role with the controllers to help. Our brains just kind of realize that feedback and realize the events happening in the game. But I imagine also sound is playing a big role in that experience too. With VR, you have what’s called binaural audio, where you have this sort of 3D audio experience being able to rotate your head at a 360 degree angle. Or even if you move forward in the game, your surroundings, the audio is going to adapt. So things that you may have heard that were right next to you as you move forward are suddenly further away in the distance. And you hear it from that direction. Can you talk about how sound design has played a role in this process, both from just the base game, but also for the accessibility of the game?

PETER GALBRAITH: Sure. So sound design is incredibly important to our games. And I would say incredibly important to VR as a whole. We, as people, are very good at determining where a sound is coming from in our environment. We may not be incredibly precise, but we’re pretty good at it. And so it’s very important for sounds to be placed where they are in the environment so that you know where to look, where to interact. And this is especially important in this update that we’ve been working on because there’s so many objects. And being able to indicate to the players where they need to be interacting. It’s just very key to getting people to the right locations. And then using the volume of that sound. So if it’s far away, it’s going to be a little quieter. If it’s going to be up close, it’s going to be louder obviously. And that really helps give a sense of where an object is in the environment.

JAMES RATH: Imagine. Go ahead.

PETER GALBRAITH: Go for it.

JAMES RATH: I imagine that, especially true today, but with nontraditional VR games, sound design has definitely become a more I guess budgeted component of the game. With the fact that a lot of players now play at PCs with headsets on, whereas maybe 20 years ago, most games were being played on console with just mono audio through their old SRT or through their old– is it SRT?

PETER GALBRAITH: CRT.

JAMES RATH: CRT. I’m getting– see, accessibility is on the mind. Captioned files. But yeah. Their old CRT TVs. And so could you tell me a little bit. Just as a developer doing VR games, has there been any difference with sound design or is it even more crucial or important than, say, a non-traditional VR game or non-VR title?

PETER GALBRAITH: I think it’s– I would say it’s more important now than ever. I think there are some additional challenges of in a traditional game, when you’re playing audio from a– you can play audio so that it’s coming out the stereo speakers are coming out the headset. It’s not necessarily as important to know all the exact positioning because you’re not necessarily in it yourself. But when it is physically your body that you are moving through the space, it can be very important.

JAMES RATH: I think what you brought there was really important to note with that, is you’re in a first person point of view, like quite literally. Like not just from that visual perspective like a lot of console or PC games, but you’re in control of the body from all fluid movements. So no, thank you for just clarifying, just kind of adding some more context to that too. Just for those who haven’t experienced VR, I think it’s important to really help illustrate the difference of gameplay from a player’s perspective.

PETER GALBRAITH: Absolutely. And with regard to some of the work that we’ve done here, because of how important it’s been, we’ve had to do a lot with prioritizing what audio is important for the player to hear in these contexts. So we end up doing some ducking of audio to make sure that the most important audio is getting heard by the player. So in this case, that would often be the text of speech that is being triggered. Because that’s– obviously, the player is trying to hear information about the world around them and use that effectively. Additionally, we do some information– or we do some prioritizing and canceling audio as well. So in the video that we showed a little bit ago, there was a moment where I move from one to– From one object to the other while it was still speaking. In that moment, the previous audio was canceled and the new more relevant audio was played. If we didn’t do that, it would become cacophonous and just really hard for players to understand. So it’s really valuable to make sure that you’re prioritizing the right information.

JAZMIN CANO: In addition to that, as well– oh, sorry.

JAMES RATH: No, no. Go ahead.

JAZMIN CANO: Adding to the counseling audio there, we also allow players to cancel any of the text to speech from objects that are being described when they want. So there’s agency there as well. And I got to say we learned a lot of this in our playtest. Early playtest did teach us that it’s better to duck audio, cancel it here and there. Peter described it as a cacophonous experience. And it’s hard to describe it, but it’s not great to have lots of audio overlapping. Because when you have more than one voice speaking at you, it’s hard to understand . In addition to that, having world sounds and music, and maybe NPCs speaking, it adds on. And most often, it’s hard to understand anything that was said in that moment.

JAMES RATH: Yeah. I think that that’s really important to remember too, is some players– when we think about accessibility, it’s a very optional or customizable experience for everyone because everyone’s needs may be a little different. Some people’s gameplay experience might be a little, on various different levels than others. And even if someone maybe comes back to the game months later when a new DLC pack is available, they might want to get re-familiar with the controls or some of the protocols or practices with the text of speech. So I think that’s always important to be able to cancel things out when you want, but have it there to remind you what’s going on when you need it.

PETER GALBRAITH: Absolutely.

JAMES RATH: Although Cosmonious High is a game, do you think that these accessibility options that your team has been building could apply to other genres or applications by chance?

JAZMIN CANO: Yeah, totally. This isn’t specific to our game or VR games exactly. It could be for other VR experiences. So like a traditional screen reader, we’re building this so that people can hear descriptions for objects. There’s confirmation of when something was grabbed and released. And that could be related– sorry. That could be compared to when someone selects an option on a traditional screen reader on a 2D screen. And they’d have something was selected. And then, again, we just talked about canceling audio and then balancing that audio. All of this could be applied to different experiences, not just VR games.

PETER GALBRAITH: Yeah, absolutely. I definitely think that this is something where we are trying to design the interactions and see in these– part of the reason Owlchemy has been doing these at the end of our products is because we’ve been using that as, hey, here’s a dense full world. And we are interested to see, OK, how do we make this work for everything that we can in this world? So it’s really important that we are supporting a variety of different interactions and different– just things in the world. I mentioned that there are differences between objects that you can pick up and grab and objects that are sort of locked to the environment. Different interactions are going to require different approaches. But by using some systemic design and trying to establish some common practices across all of these, we’re hopeful that we can find some sort of commonality that can be used in any type of VR, XR, and maybe even AR game or application in the future. I’d love to see this idea of pointing at an object and getting contextual information brought, about it, brought to an augmented reality environment or other XR spaces as well. So hopefully, what we’re doing here in the designs that we’re putting forward will be helpful to designers of all types.

JAZMIN CANO: Oh, can’t hear you, James.

JAMES RATH: Sorry. Muted for a second because a loud motorcycles going by. I’m gonna back up real quick. Totally agree with you. And I love that you said, just being able to use other applications, I think is such an important thing from a user perspective. Because being able to just boot up my headset, and if I had these similar features to even get into the game or find my way around the UI menus of the platform that I’m using, I think that’s really important and helps us set a standard for the developers as a whole, that are creating for that platform. But right now, you guys are really leading that. And with that said, like if another developer sees the work that Owlchemy Labs is doing and they wanted to make their game or application more accessible, what advice would you have for them or how can they get started?

PETER GALBRAITH: Well, the first thing I would say is start early. Something I’ve learned from working on these is that it’s challenging to integrate these later on in your projects. And the more solidified your stuff is, the harder it’s going to be. I think it can still be done. Again, we have a wonderful team that’s built a great game with wonderful bones, so many things. We were actually able to hook into some of our other systems to do this. But some things had to be layered on top. So thinking early on about what you can do is better than trying to retrofit at the end.

JAZMIN CANO: One example of this that we can point to in Cosmonious High is early on, the objects were named and had descriptions. So that when we started this work and you could start scanning the objects around and hear their descriptions, it was already there. So we didn’t have to manually, later on, go and type everything. And that’s really important because it’s just like all text on media and traditional 2D screens. We have to have it on everything. And it was already there.

PETER GALBRAITH: Yeah, absolutely. And again, I mentioned that we had great bones in our game to sort of build on top of. But at the same time, if you’re a developer looking to get into this. And I’m saying, get into things early. Understand that doing something is better than doing nothing. You don’t have to be perfect the first time. You don’t have to do every feature if you can’t. Anything that you do that helps someone is a step in the right direction. So as I think we mentioned earlier, right now we’ve been focusing on trying to get users that are blind and low vision to be able to complete our game. I think we’re doing really well on the low vision and I think we’ve started the process for users with complete blindness. But I think we still have room to go. And I think we still have room to go for users with low vision. But I think by doing something and getting it out there, we’re going to enable the collaboration with other developers. And please take our designs and use them in your game or application. See if they work for you. Let’s work together and try and come up with something that everyone can benefit from.

JAZMIN CANO: Absolutely. And you can trust that our designs, we have real feedback from real people who have played tested it, who have low vision and blindness. And what we’re putting out is just things that we’ve kind of co-collaborated on, that we know are working, they’re successful so far. And in surveys after playtest people have said that they would recommend it, they would use these features. And it just was great feedback overall. So you can trust that what we’re putting out there is really helpful for this game.

PETER GALBRAITH: And I mean I think it’s to be said also do your own play testing. Reach out to real people with disabilities. Reach out to friends. Reach out to– if you’ve got a community around your game or application, I’m sure there are people in that community that do have accessibility challenges and would love to have those concerns be heard. And then work with consultants if that makes sense for you as well. People that are a part of these communities. And make sure that the solutions that you’re implementing are actually benefiting them. And I think that will get you a long way if you’re just making sure to listen to the feedback of the people who actually need the things you’re creating.

JAMES RATH: Awesome. Thank you. I love to hear about how collaborative and supportive the whole developer industry is. Just as a whole, I know from my experience as a accessibility consultant from other developers, everyone is there to support one another. And we’re still trying to figure all this out. Like that’s the thing is, it’s great that VR is now making these steps to be more accessible. But even the traditional games are still making huge strides and still have a lot of progress to go. So again, thank you for the work you guys are doing. Is there any closing comments or any things that you guys just want to make note of?

JAZMIN CANO: Yes. For developers, do accessibility work. It’s important, it’s the right thing to do.

PETER GALBRAITH: Yeah. I mean I echo that sentiment. And again, go out and find the communities around. If you’re interested in XR, VR, AR, those things, the XR association, the XR access community. So many more are out there and are just wonderful people that are trying to share experiences and trying to share this knowledge so that we can establish this stuff. Because VR is still early as a medium. And if we can establish good accessible design practices now, we’re not going to be excluding people throughout the life cycle of this hardware and medium as a whole. Hopefully, we’ll see this become integrated into what is common and expected from VR.

JAMES RATH: Awesome. Thank you. Again, I’m excited to continue to see how this evolves in this industry.

PETER GALBRAITH: Thank you, James.

JAZMIN CANO: Thank you.

[MUSIC PLAYING]