For Most Mobile Phone Users, Accessibility Is Spelled Android

Nearly three quarters of mobile phone users in the world use phones built on Google’s Android operating system, not Apple’s iOS on the iPhone. For people who are blind or have low vision the key app is Google’s Lookout, which draws on the vast resources of Google’s AI infrastructure, including its computer vision database and Google maps. How is Google approaching the huge accessibility opportunity Lookout represents?

[MUSIC PLAYING]

WARREN CARR: Thank you so much, Will, for giving us an opportunity to come here today. And today, I’m here with wonderful people from Google. And we’ll be talking about Google and all the wonderful things that Google has been doing.

And now I would like to get started by talking about Google in general as a company and Google’s attitude or approach toward accessibility. How does Google in general view accessibility?

EVE ANDERSSON: I’ll take that question. Hi, everybody. My name is Eve Andersson. And I lead accessibility for Google. And accessibility is really core to what we do as a company. It’s even part of our mission. Our mission is to organize the world’s information and make it universally accessible and useful. And that really does mean for everybody.

And so we consider accessibility to be part of everyone’s job. There are many people across the company for whom accessibility is their full-time job. But even for those who don’t work on accessibility full time, everybody who works on products goes through accessibility training when they join the company. Because we don’t want there to be people doing it incorrectly from the beginning. And then others have to go in and fix it. That’s just not a good way to make accessible products.

And we know that it’s really important to incorporate accessibility at every stage, from design through development through testing and release and user research and all of the above. And so it really has to be integrated into everything we do.

And I also want to point out that we don’t just think about accessibility as a compliance thing that we have to do. Because I think some companies can get into that trap. But really, accessibility is about so much more. It’s about usability. Not just can a person use a product, but is it enjoyable to use a product?

And it’s also about innovation. And I’m really happy we have two members of our Lookout team here who will talk a little bit about what they’re working on later on. But that’s a really great example of innovation and not just compliance.

And the last part of the approach that I really want to call out is that products can only be good if there is a culture in which people are included. It’s the people who build the product. And so it’s really important to have people with disabilities who are not just users, but are part of the entire process of developing the product.

And so that means hiring inclusively. It means making sure that once people are hired that they’re able to be productive and included members of the team through tooling, workspace, culture, and so on. And so I think all of these pieces fit together into this whole. Thanks for the question, Warren.

WARREN CARR: Thank you so much, Eve. And I happen to be one of those blind people that benefit from what Google is doing. And staying on the topic of the subject of accessibility, what are some of these products or features that Google has when talking about accessibility, both about blind people or visually impaired? What are some of these features or products that blind or visually impaired people may find of interest that’s coming from Google?

EVE ANDERSSON: Well, hopefully all of our products are of interest. Because we try really hard to make them all accessible. But let me actually answer your question.

So you can think about technology for accessibility in a couple of ways. One is digital accessibility. So making the digital world, the digital products that we use, work for people with disabilities.

But then the other aspect of it is physical world accessibility. So how do we use our digital technologies as a tool for making this real world that we all inhabit more accessible? And I think there are really giant opportunities in both of these areas.

So in terms of digital accessibility, it’s making the product accessible. It’s making the operating systems accessible also. So two of the most popular operating systems that Google owns, one is Android for phones. And another one is Chrome OS for computers, for laptops.

And so both of these have a number of accessibility features built in, like screen readers, magnification, et cetera. Just part of the operating system, not a separate piece of software that you have to go and buy and install and all that.

And then making all of our products work, we make sure that they work well with screen readers. We make sure that they work well for people with low vision also. So contrast, ability to do magnification, and so on.

And then in the physical world realm, you’re going to learn more about Lookout, if you haven’t– I know that you, Warren, have tried it, of course. But for those of you who haven’t, you’ll learn more about Lookout. But that’s one of a set of things that we’ve developed for physical world accessibility.

Another one, which is much earlier stage, is called Guideline. That is for people who are runners. And it uses AI to help a person be able to follow a physical line on the ground and run completely independently.

Another example of physical world accessibility is a feature in Maps, which will give detailed voice guidance. So it was developed specifically with blind users in mind to just give people a little bit more information about what’s around them in their environment.

WARREN CARR: Great. Thank you so much, Eve. There was something you mentioned earlier that really caught my attention. And that has to do with the inclusion of blind people or people with disabilities in general with what you guys do.

And there’s a saying that goes on within the blind communities or disability communities that goes something like, nothing about us without us. And so in other words, if you’re putting out products or services or features out there without voices from the disability communities, as far as people with disabilities are concerned those products or those features or whatever is of no significance to them.

So what is Google’s approach in making sure that the voices of these people with disabilities or voices from the disability communities are being heard in making these products or features that are beneficial to us?

EVE ANDERSSON: Yeah, that’s such a great question. And I agree completely that it’s so, so important to include the voice of the user at every stage. And yeah, as you mentioned, I did bring up hiring, inclusive hiring, earlier. And that’s a really important piece of it, but it’s not the only piece of it.

And the truth is even if we hire people with disabilities, they’re not going to necessarily be representative of the world at large. You have to get more voices out there, more types of users, users who are not software developers.

And so we have a number of programs to make sure that we really are integrating a diverse set of user voices. For example, we do user research on a regular basis. It’s a best practice to involve real users during ideation for products, or testing out new features, or making sure that at the end the product actually performed as you expected it to perform based on all of the feedback that you got during the process. And so there are places where users can participate in research throughout the processes.

And in addition to the more general user research, we also have a standing panel of trusted testers who have been getting early access to Google products and giving us really useful feedback and ideation, not just feedback on existing things, but ideation just across the board. And that includes quite a few users with disabilities who we recruited specifically for this effort.

We also have a disability support help desk which is completely free, where users of assistive technologies or other users with disabilities can contact us in a number of ways. There’s the standard support channels that most companies have, like email and chat and phone. But we also do ASL support, and we do support through Be My Eyes, which, of course, is a really popular product out there for people to get sighted assistance if they so choose.

And so people are able to contact us directly through this platform. And it’s a two-way street. One, we are, of course, providing assistance to the person who asks for it. But at the same time, we’re learning. And we’re making sure that we’re seeing trends, uncovering any issues that might exist, getting ideas.

And we’re feeding those back to the product teams so that even after we launched, that’s not the end. That’s just a stage in product development. And so we really have these open lines of feedback.

And then one last channel I’ll mention is through our partnerships. So we have partnerships with a lot of different organizations around the world, from big national organizations through small nonprofit community organizations. And that’s another way in which we’re able to get really structured, detailed feedback. Sometimes it’s even different than the feedback that we’re getting from individuals, because we learn about systemic things that maybe an individual wouldn’t necessarily notice or experience themselves.

And so this has been so helpful for us as we develop our roadmaps and our priorities and our individual products and features.

WARREN CARR: Thank you so much, Eve. That’s wonderful. And let’s talk a little bit more. I’ve been using Google products for quite a while now. And I’m looking back at the past five or so years. I notice a lot of things have changed a lot, especially with reference to accessibility. How have you seen accessibility changing over the last five years in general in the industry?

EVE ANDERSSON: Yeah, I think it has been an industry-wide change. And we’ve certainly seen a lot of that inside of Google. But it’s a reflection of the broader world, of the attention that people are paying to accessibility and the importance that they see it hold.

And I see this being both a cultural change and a change in expectations, and at the same time, a technology change. But you do see that accessibility is now often a requirement, not a nice-to-have. And not just internally, but we see that from our customers, from our users, but also the business customers that we have. It is an expectation. And I didn’t necessarily see that five years ago.

The other thing that I’m seeing is these changes in technologies, AI being a really major one. Because among the many things that you can achieve with AI, one of them is a change in modality, which I’ll explain in a second. But that can really help with some assistive technologies.

So when I say a change in modality, what I mean is that AI is able to understand different things, like speech or an image or handwriting or other things, and translate it into something else, whether it’s text or speech or something like that. And I’ve seen AI– so many strides happening that can then help with understanding in both the digital world and the physical world. That’s one of the biggest changes I’ve seen over the past five years.

The other major change where I think there’s still a lot of runway is around internet of things. And so that, of course, is connected devices of any type, whether it is thermostats or lights or fans or just really anything. And while that wasn’t built for accessibility in the first place, because the systems that control this are accessible, because it’s built into the operating systems, that means that you’re able to then use these modality changes to give input.

So you can use your phone to operate your thermostat. And because your phone has accessibility built in, you don’t have to see your thermostat, or you don’t have to be able to use your hands. You don’t have to twist it. And so it just offers a lot of opportunities.

So those are a few of the examples of changes I’ve seen over the past five years. We also have two team members here who work on Lookout. And maybe I’ll just briefly introduce them. Scott Adams is the product manager who works on Lookout. Andreína Reyna is the tech lead, the head engineer who works on this.

Andreína or Scott, would either of you like to talk about any changes that you’ve seen over the last few years?

SCOTT ADAMS: Eve, I think you summed it up very well and have probably seen more than I have. I am very excited by just the advances in technology. It still feels very early days for everything we’re working on. But you can see things become possible that were much more science fiction or wouldn’t it be cool if. And now it’s like, oh, we can actually see over the horizon a little bit. And it’s coming.

So I think, overall, it’s a very exciting time to be working in this area.

EVE ANDERSSON: It sure is.

WARREN CARR: That’s so good that you brought that Lookout in there. And most especially about AI, which is playing a very important role in our lives today. I happen to be one of the people that benefit from that. My thermostat– finally, a blind person is able to control his or her temperature in the wintertime or the summertime. And I’m so proud of that. I’m very happy using my Nest Thermostat.

Now, you mentioned the subject of AI, which is very important. And we talk about Lookout. For Scott or Andreína, would you guys mind describing Lookout to someone that probably is hearing us for the first time and has not heard about Lookout? What is Lookout?

SCOTT ADAMS: Sure, I can take a swing at that. So Lookout is an app for Android phones that can be thought of as a talking camera. So we basically use the smartphone camera to recognize objects in the environment and then announce those to the person using Lookout.

So for example, this might be reading short text or a long document. Or it might be recognizing packaged food, like a can of soup or a box of cereal, or identifying currency, like dollars or euros or Indian rupees. We can also do things like recognizing objects in a room. So if I go into an unfamiliar room, we can say, desk, couch, chair, and so forth.

And the idea here is to give people who use Lookout a better way to interact with the world around them and become a little more accessible for that physical world.

WARREN CARR: Talking about Lookout– and I know Lookout is primarily for the Android operating system. Now, you and I know that Android comes from probably literally hundreds of manufacturers out there. And therefore, it becomes really a challenge, and a very huge one for that matter, in trying to ensure that Lookout works across the board of the different phones out there from different manufacturers. And I know this must be a huge challenge for you guys.

How do you go about making sure that the phones that qualify have the ability– or that Lookout is able to work across these different phones? What are some of the requirements? Does it require a certain version of the operating system? Or how do you go about managing this? It’s a huge task.

ANDREÍNA REYNA: Yeah, great question. I can take this one. So yeah, definitely working on the Android ecosystem can be a big challenge. And it’s one of the bigger challenges that we face when building Lookout. And this is one of the reasons why we’ve actually had a gradual rollout expanding to more and more devices over time.

Even just a few months ago, we released a new set of devices that are supported. So this is still an ongoing process, even though at this point, we only have a very small number of devices that are not supported, most of them on the very low end range. And we do try to make sure that all of our features work on all of our devices.

So we think that there’s a group of features or baseline of features that are so important that we really wanted to make sure they worked on all devices. And so we have been testing and doing this gradual rollout to make sure that the features that we have are supported in all of our devices. And this may not be true for the future. But so far, this is how it works.

And then the last thing I’ll mention is that there’s also another trade-off to consider, which is that Lookout runs most of its processing on device. And this has a lot of benefits. You don’t need internet connection, for example.

But it also means that the device itself is more important. So if you have a faster phone, it will definitely work a little bit faster. And so these are some of the technical considerations that we think about when we make a trade-off between having a feature that works in the server versus a feature that works in the device.

WARREN CARR: Thank you so much, Andreína. And I really like the part that you talked about the fact that Lookout works offline. And that’s one of the key ingredients for me when it comes to using an OCR package or any app that is found within that class, that privacy. Because just like anyone out there, blind people, we have need for privacy.

And it is very important that I am able to scan something offline. There are certain things that I don’t want someone seeing out there. And I find Lookout to be something that scans my things offline, whether I’m connected to the internet or not or whether I have airplane mode on or airplane mode off. So it makes it important that I have that capability.

Now, talking about AI and still on that subject of Lookout, are there any roles that AI plays in some features with respect to Lookout that we currently have or not?

ANDREÍNA REYNA: Yeah, so I can take this one as well. Definitely AI is a huge part of Lookout. And in particular, computer vision, which is the field that intends to get meaningful information from images. And as Scott explained before, Lookout works by taking images from the device’s camera.

And then these images are processed in a bunch of different ways. So for example, depending on the Lookout mode that you have selected, we will run different computer vision models so that we get the information that you’re interested in. So for example, in text mode, we may run an OCR model, which is the one that will identify the text. And then we can read that text out loud.

But in other modes we might run different models. And sometimes we even run multiple models at the same time and combine the information so that we can give the user the most useful information.

WARREN CARR: Now, going back to users of Lookout or to any other app out there, I know that you guys like hearing from users. Now, what are some of the features that you think your users are telling you that they enjoy the most, especially when it comes to Lookout? Because Lookout has several features in there. But what are some of the features in Lookout that people tend to use the most that you hear feedback on?

SCOTT ADAMS: That’s a great question. I can answer that a few different ways. So I’d say the most commonly used features will be those around text. Text we know is very important. It’s very information-dense. And it’s often not accessible.

So things like using text mode or document mode to understand printed text, to look at displays. We may not have a screen reader. Those are our most commonly used modes.

But we also think about not just the frequency of the use, but also the importance of the need. So let me give you an example there. So we have currency and food label mode also. And we don’t expect someone to be using those modes 20 times a day. But there are moments when those modes can be very, very important.

One user shared a story with me where her partner had come home with the groceries and then had to run out. And she had multiple bags of groceries. And they had been in the car for a little while. So everything was kind of cool, but it wasn’t clear what had to go in the freezer, what was supposed to go in the fridge, and what could just go in the cupboard.

And the user says, oh my gosh, how do I do this? Am I going to put the ice cream in the cupboard and the box of cereal in the fridge? What do I do?

And she ended up using food label mode to go through and identify each item and put it in the right spot. So that was one of those moments where you might not think, oh, this will be how I use Lookout. But then having that functionality gives you the option of independence to kind of do those things on your own and hopefully do them a little faster and a little more easily.

WARREN CARR: Thank you so much, Scott. Now, it’s interesting that you talked about the currency, the subject matter of currency. Of course, I always argue that I don’t have money in my pockets all the time, you know. [LAUGHS] So I don’t use that feature a lot of times. Seldom I do.

But now there are some things that people are not aware of. And that is true with just about every app out there. Sometimes there are great features out there found in such a particular app, but people are not aware of. Are there any things or features in Lookout that people may not know of that may be of benefit to them?

SCOTT ADAMS: That’s a great question. And you might chuckle about this one. I think the one where people are still discovering it is actually one where we had a pretty big release about it recently. And that’s being able to have handwriting recognized.

It’s actually built into document mode. We got some articles about that, which we’re very thankful for. But we made the decision, hey, we don’t want to just keep adding modes to Lookout unless we have to. We want to be intelligent and thoughtful about packaging those together where they’re related.

So we said, OK, let’s put handwriting recognition within document. So if I get, let’s say, a greeting card, I can both have the printed text on the outside identified and then open the card and have the handwriting inside identified. And that makes a much smoother, easier experience.

There’s only one mode. It’s already familiar. But I’m not always sure if people realize, wow, it can actually do both now. So I think that’s still kind of being discovered by some folks.

And then along the same lines, you just mentioned currency. Like, hey, maybe you don’t use currency very often. And that’s another thing that we think about. OK, how can we make it a bit easier to configure Lookout to fit each user better?

So we have a way to actually say in settings, I want to manage my modes. You know what, I never carry cash. I only use credit cards. I’m going to go ahead and hide currency mode from that menu. And now you have fewer things to navigate through.

And then the other option we also have is being able to adjust the speed and the tone of the text-to-speech. So when Lookout announces something, how fast does it speak? And what’s the pitch of the voice?

And this can be useful, because someone may be using a screen reader that speaks very, very fast. And they want to get Lookout results a little bit slower so they can parse those and think through the next thing they want to do. So those are two items that people often find them and are happy they exist, but may not be immediately aware of them.

WARREN CARR: That’s very interesting that you mentioned that fact about speed, speech rates, and things like that. Because sometimes we hear people complaining about certain apps that are in this category that may be too fast for them to understand what is going on. And it’s interesting that Lookout, one could control that.

And that makes all the difference, because I do not need to have the same speech rate that I have on my screen reader if I’m trying to read something and carefully pay attention to what is going on. So I’m glad you mentioned that.

Now, let’s talk about another part of Lookout. Are there any advances in AI that you think would actually even make things a little bit better for Lookout in the future and things like that, since AI is playing a very important role?

ANDREÍNA REYNA: Yeah, I’m definitely counting on it. I think advances in AI are very exciting. And there’s a lot of things that we can expect in the coming years. In general, I think we’ll be able to see more powerful models. So models that can do more things.

And I’ll give some examples of this. The visual world is very rich, and people want a lot of information. But right now, we have these very specific modes that just kind of give you one information at a time. So being able to combine and augment all of this information I think would be extremely useful.

Another way to think about it is also on internationalization. So making Lookout work in different languages and different countries. So right now, it usually takes a conscious effort to– for example, if we have an OCR model that works in English, characters in English don’t look anything like characters in Hindi or in Japanese.

So it takes some conscious effort to make the same kind of features work in different languages. And the same happens, for example, in currency mode where each country has different currencies. So things might be slightly different. And we want to make sure that it works for everyone. So being able to generalize much more easily, so having models that can just– you kind of train it once and works for all those kind of things would definitely help us scale much faster.

And then the last thing I’ll mention is that we do have some of these more powerful models already available on servers. But like you said, our users really like when things work on device. So I do think we will continue to see improvements on what can be done on device. And we will continue to see both more powerful models and more accurate models that work directly on the device.

WARREN CARR: Thank you so much, Andreína. And that’s an important part of the program. Because I know a lot of my friends that are from other parts of the world don’t currently have Lookout supporting their languages. But with AI improving things, it’s likely that we’ll see these languages or these other languages getting the support that they need. And that’s a very good one.

And I remember I gave Scott a little bit of grief on our podcast, the Blind Android Users Podcast, when we talk about handwritten material. Because it’s something that students would find of great use to them.

Now, from the Lookout team’s viewpoint or perspective, what are some of the challenges and/or opportunities of both blind and low vision users that you guys see? And any of you, either Andreína or Scott, any of you can talk about that either from a technological viewpoint or from a usability viewpoint.

SCOTT ADAMS: Sure, I can talk about the usability side and then maybe let Andreína give the tech perspective. I think there are a couple really exciting and challenging kind of problems to work through. I hesitate to say problems, because they’re really enjoyable areas to work on. But I’ll summarize those.

So number one, we know that often people who use Lookout want both the ability to get a very kind of wide, shallow summary of, let’s say, an environment or of a set of text and also kind of narrow down and go very deep. So as an example, think of, let’s say, a newspaper, where there are multiple large headlines and then many columns of article text that’s much smaller beneath those headlines.

And how do we let someone both understand what those headlines are and then decide, OK, it’s that fourth headline that I really care about. Don’t tell me about any more headlines. Now let me kind of deep dive and zoom into the article beneath that.

How do we kind of give them enough information so they understand how they’re oriented? And then once they feel like they have enough, let them say, OK, I want to go and focus just on this and get rid of everything else. So that’s kind of one question that we really look at and are doing a lot of work on.

Another one is just around the camera itself. Since this is computer vision, the image going into the camera matters a great deal in terms of the quality of the results you can get coming out of it. So how do we help users take really sharp, well-lit images, make sure the focus is good? And that if there are multiple items in the image, we know which item the person really cares about. And what if only part of that object is visible? Is there a way we can let the person know that and perhaps coach them so they can get better results?

And then the last point, which I alluded to bit earlier, is around the interface also. How do we make it very easy to find and use the functionality that the person wants without having too many options, too many settings, too many things to navigate through? And then obviously, for people who both use a screen reader and for those who don’t, what’s the best interface for their particular situation?

So those are kind of the big three things that I think about. Andreína, from the tech side, what’s top of mind for you?

ANDREÍNA REYNA: Sure. So as you can imagine, there’s a lot of relation with the things that you said. But I can mention a couple more. So for example, on the image quality side, it’s definitely one of the big challenges that we still see. The way traditional computer vision models are trained and the images that they use, they don’t necessarily look like the images that we get in Lookout.

And so that means that there is a slight mismatch. And that means that maybe the quality is a bit worse because of it or that we need to do extra work to make sure that we’re getting the results that we want. So bridging those gaps I think is still an important problem to solve.

And this can be done in multiple ways. It can be done both by improving the computer vision models themselves or the data that is used for training them. But it can also be done, as Scott mentioned, maybe as a way to preprocess images before we send them to the model or even on just the usability side on helping the user to use their camera in a way that would give us better quality images.

And on the feature side, Scott’s mentioned text. And for every single feature that we have, even if it’s launched, it doesn’t mean that we think it’s done. Every feature has room for improvements. So in text recognition, for example, we know that there are sometimes the small mistakes. And we wish they didn’t happen.

But even beyond that, we would like to work, for example, on making sure that we are reading things in the order that makes the most sense. Especially if you have some form or table, we know we don’t really do a great job of it by now. And as Scott mentioned, if there’s a lot of text, then trying to narrow down what is the most important things for the user or giving the user a way to tell us what is it that they want.

And just like I mentioned before, we have an endless number of feature requests from users. So those advances in AI that I mentioned before, giving us more powerful models that can give us richer information about the world. We don’t want to just say that an object is there, but we want to give maybe some more description about the object or what’s around it and a few other things like that. So getting those rich descriptions would be also on the top of my list.

WARREN CARR: Thank you so much. And thanks to Eve Andersson from the accessibility department. Thanks to Andreína, who is a software engineer, and from Scott, who is the Lookout head. We’re looking forward to seeing you guys next year on Sight Tech Global. And we’re thankful to those of you out there that are watching Sight Tech Global.

[MUSIC PLAYING]