Unlocking Human Potential: Universal Design and AI with Meta

Join Agustya Mehta, Director of Concept Engineering at Meta’s Reality Labs, to reflect upon his lived-experience with technology and how it has enhanced everyday life for the blind and low vision community, and discuss how AI-powered wearable technology can address some of the many challenges that have yet to be tackled. We’ll cover emerging trends in this space, Meta’s exciting portfolio of AI-powered Wearables, and how developers can create groundbreaking experiences around independence and inclusivity for Ray-Ban Meta glasses and beyond.

[MUSIC PLAYING]

VOICEOVER: Unlocking Human Potential: Universal Design and AI With Meta. Speakers: Agustya (Gus) Mehta, Director of Reality Labs Hardware, Meta. Moderator: Karae Lisle, CEO, Vista Center.

KARAE LISLE: Hello, everyone, and welcome to Unlocking Human Potential: Universal Design and AI at Meta. My name is Karae Lisle, and I am the CEO of Vista Center. We are a human services nonprofit here in Silicon Valley, and we are the producers of the Sight Tech Global Conference. We’re very excited today to have, uh, speaker Agustya Mehta, director of concept engineering at Meta’s Reality Labs. And this session will be a presentation and a discussion that will reflect on how Meta’s technology has enhanced everyday life for the blind and low-vision community. Gus will share how AI-powered wearable technology can address some of the many challenges that have yet to be tackled for our community. And so I’m gonna turn it over directly to Gus and let him get this session started.

AGUSTYA MEHTA: Thanks for the really kind intro, Karae. I’m really excited to be chatting with you today, um, and honored to be here. Uh, I’d like to talk a little bit about what I do at work, um, but I’m not just gonna come from the perspective of a engineering leader at Meta, um, but also I wanna talk through my own personal background, ’cause it’s really informed a lot of what I’ve done in my career. And I think it’s also given me a unique perspective that is probably shared with many of you on the real potential the technology has to improve lives of people all around the world. Um, and something I think is often missed by a lot of people in the mainstream technology world.

So starting off a bit, my name’s Agustya. I also go by Gus. Um, I lead a team called Concept Engineering here at Meta. We build early prototypes of our AI devices, augmented reality glasses, smart glasses, other wearable gadgets. And it’s really fun. My technical background is physics and electrical engineering, and I get to put that stuff to use every day. Um, before Meta, I’ve worked on a number of different types of consumer electronics, from the Xbox and the iPhone to the HoloLens, um, to components of self-driving cars and data for autonomous vehicles. And a big part of why I’ve chosen to do those things is not just ’cause they’re fun, cool tech, but because of my own background and my history with vision loss.

So I’ve got a fun picture here. For those who, uh, can’t see, I’ll describe it. It’s- it’s a shot of myself and my mom and her guide dog, a full-length German shepherd named Amber, um, from about 30, 30-something years ago. So I’m a lot more fun and, um, cheerful and good-looking in that picture than I am today. But generally speaking, the same person. And I show this picture and I tell the story not just ’cause it’s, like, a fun, adorable picture, um, and it’s got an amazing, cute dog in it, but because it’s an illustration of, um, I think the state of the art in navigation, um, that really is sort of still the way things are today.

So in this case, we’re aboard the USS Nautilus. It’s a naval submarine that has since been turned into a museum. And so you can imagine navigating such an environment without sight. Even with a cane, overhanging obstacles and portholes will bonk you on the head. Amber’s able to navigate it pretty well, um, even though they don’t necessarily train in submarines when, uh, when guide dogs for the blind, uh, trained her, she was a very clever dog. I think it’s an example of an intelligent assistant, and it’s still the state of the art. It’s canine intelligence is the word I’d use.

Um, funny story, by the way. My wife’s name, who I met, um, many, many years after this picture was taken, uh, is Amber, um, which causes, uh, endless, uh, endless jokes within my family and, and a little bit of teasing, but that’s okay, I can handle it. Um, so I think that we’re at a point in time with the way technology’s advanced where despite how cute and amazing guide dogs are, they might not be the state of the art in the next couple years. I think that the combination of wearable technology and AI capabilities is gonna make intelligent assistance something that’s scalable and available to everyone.

So I’m gonna take a little bit of a stroll back in time for a bit. Um, many of you have lived through this, so I’m probably preaching to the choir in many of these cases, but I still think it’s a story worth sharing. Growing up and seeing mom interact with technology, I saw firsthand many of the ways in which she was empowered by devices and products, and also many of the ways that she was maybe left behind or where there was opportunity for improvement.

Um, if you go back in time to, like, the ’60s and ’70s, a lot of things about society were markedly worse for the blind and low-vision community. Now, I don’t have to state all the changes that have been made in a positive direction, but suffice to say if, if mom wanted to see the chalkboard, uh, even when she had some vision, it was through the goodwill of a teacher that they might put her in the front row, but there was no law protecting her right to do that. And in many cases, the content of a class was just not available to her, and it was, there was no recourse. So certainly many things have gotten better in society since then.

But there is one aspect on the engineering and product design side that I think maybe we’ve lost. So pictured here is a rotary telephone. Um, for those of you who remember what it actually was like to dial a phone, that’s where the word comes from. It’s also an AI-generated picture, which I like to throw little Easter eggs in my presentations. So this, this phone in particular has more than 10 digits, and the digits are labeled with gibberish right now, but it’s a, it’s a rotary phone nonetheless.

I throw this example up there because even though this is a pretty old technology and it wasn’t designed expressly with the blind and low-vision community in mind…… it’s a telephone that’s probably a lot easier to use for someone who can’t see than many modern devices are, because when you put your finger on the dial, you feel it. You can hear the click, click, click as you turn it. And if you can see, you can see it. This design philosophy of trying to make something work well across the senses, maybe if I generalized it a bit, I call it universal design, is a thought process where if you make stuff good, no matter what kind of mechanism you use it by, you make it good for everyone. You not only make it accessible, but you made a better product.

Now, back in those times, like I said, maybe things were worse, but my mom was pretty adaptable and she was able to use early computer terminals. She wrote code with punch cards, early speech synthesizers let her understand what was being put in and out of the command line. And her shortwave radio, which was my dad’s, uh, gift to her when they got married, um, which had physical buttons and physical controls, was her connection to the world. She often got news before the rest of us did. I still remember waking up and hearing her talking about how they cloned a sheep. We didn’t believe her, but it took two days for that news to hit the US. She heard it on the BBC World Service, like, live as it was happening.

So, the story of technology going from that decade forward to maybe the ’90s, in the mainstream, it’s often a story of making computers accessible to the masses. I have pictured here a shot of an old desktop computer with a blue screen on it. I did a lot of prompting to the AI to get it to give me a good blue screen of death. And this era, while it did realize the vision of putting a computer on every desk in every home, uh, that Bill Gates had, it was fraught with a few design decisions that left people like Mom behind a little bit.

When we moved from command lines to the graphical user interface, we made computing a little bit more accessible for non-technical audiences, and that brought computers to everyone with an asterisk. In Mom’s case, graphical user interfaces were not initially implemented with people who couldn’t see in mind. There were early screen readers, but they were clunky and generally didn’t work well with a lot of content. And as we ventured into the era of the early internet, shortwave radio, for example, stopped being so much of a thing, and my mom went from having independence around getting news to now relying on her, uh, non-canine family members to read it to her from the computer. So, certainly some steps forward, but I think in the pursuit of the minimum viable product, as we so often like to say here in Silicon Valley, and in the pursuit of moving fast, we maybe, uh, as an industry, didn’t think about how many users have needs that go beyond just the use case that maybe the designer would use if they were a consumer.

So, I now venture into the next decade, the late 2000s, early 2010s, where I started my career. And in this point in time, me, who had some vision loss myself because of congenital cataracts and glaucoma, my mom, who’s completely lost her vision, um, and other family members that are close to me, uh, really were concerned that the move from physical buttons to touchscreens was gonna be another step in that direction I just mentioned.

Now, I have to give credit to one of our competitors here, um, although they are a former employer of mine, so I, I still feel some pride in talking about it. I think that the design decisions that Apple took in releasing the iPhone, and particularly in releasing the 3GS and revision and beyond, were really profoundly good, not just for the blind and low-vision community, but for the technology landscape as a whole. Because they included voiceover out of the box as part of the product. Is voiceover perfect? No. But, in general, you can use the basic functions on iPhone without having to see, and you don’t have to pay a ton of money for it.

And the touchscreen interface, which worried me and Mom so much, actually, they thought about it as not just a mechanism to touch what you’re looking at, but also a mechanism to apply gesture input in novel ways, like being able to take two fingers and move them as a rotor, just like that rotary phone, or to be able to type and record and manner to type braille input. Now, those two feature sets in particular maybe aren’t used by the mainstream, but the design decisions that Apple took to make those work well and to make multi-touch feel really good are a part of the story of why touchscreens felt so good for every consumer when they used an iPhone for the first time, even though touchscreen technology had been around for quite some time before that.

So, with the move into the mobile space, computers became available in everyone’s pocket, and we saw the advent of some really cool applications, uh, things like Seeing AI from Microsoft, uh, Be My Eyes, other applications that utilize these mobile platforms and let people do really cool things, like have remote assistance describe what was around you, or even use AI to read text and describe images. But I think technology has taken a step that’s even more profound and interesting with the advent of wearables and with the advances in AI.

So pictured here, we’ve got, um, one of our models wearing Ray-Ban Meta glasses. And while I can’t pull off making them look as cool as she does in this picture, they are a product that I think is immensely cool, not just because it’s an iconic look and a great brand, but because of the features that they bring to the table for anyone who uses them. Now, the thing I love about this product, and I’m trying hard not to just go into sales mode here, is that they look like glasses, and they’re offered at a price point that’s commensurate with eyewear.

And I think that’s so important. I think that there are really amazing technologies that have been developed over the past few years that have implemented some of the feature sets available here. But up till now, they’ve been pretty expensive and often call attention to the user in a way that they might not want to have that attention called to them. And I think that the way technology is going, things are gonna keep getting better and better. I’m gonna talk a little bit about why.

So stepping back a bit, maybe we could have a magic wand and get everybody a guide dog. Like, that would be so cool, right? Pictured here is mom and Kovacs, who was one of her guides after Amber. Kovacs was also an amazing dog. He’s a, uh, half golden, half lab. He had quite a bit of a sense of humor. So pictured here, they’re at, uh, I believe this is Big Sur, California, near a sign that says, “Danger. Unstable cliff. Stay back.” It appears as though Kovacs is reading the sign. Um, but I hate to break it to everyone here, dogs can’t read. I, I tried really hard at this point to train him to read. They, they can’t read.

So there are some tasks that even at this time, computers are better than dogs at. And beyond that, like, not everyone wants a guide dog or can handle one. And the cost of training a dog is quite high. Uh, even if covered by charitable contributions to schools, there’s not a scalable path to deliver an intelligent live assistant, human or canine or otherwise, uh, to everyone that needs one. But the recent advances in large language models, I think have changed the game a little bit. Um, I’m gonna talk a little bit about how designing for everyone and making accessible products spurs innovation.

This ties into the story of what is gonna happen with AI, in my opinion. Often, when I give this presentation, I’m presenting to an audience that’s in the mainstream tech community. And so I use these examples to try to win people over and, and help them understand why accessibility is important. I know for this audience, you’re already on that side of the fence, which is awesome. So I don’t have to preach to the choir. But I still think these are powerful examples to share, both to give all of you ammunition when you’re trying to convince people why this is important, and also because I think it helps spark some really cool ideas.

So pictured here is a device that looks kind of like a flatbed scanner. It’s the Kurzweil reading machine. Uh, in the mid to late 1970s, Ray Kurzweil, in partnership with Bell Labs, ventured to make a device that could read books aloud for blind people. The reason I love showing people this piece of technology is, not only it’s this really cool retro look at the past, but in the process of making a device for blind people, Kurzweil invented the CCD flatbed scanner, the same scanner technology that’s, like, used in copy machines around the world. He invented reliable OCR, or optical character recognition, that could work with any font. That’s the technology that makes it so when a computer sees text, it can turn it into text that a computer can understand or grant later, not just pictures. And he invented high quality digital text to speech.

These are core foundational technologies that everyone, not just the blind and low vision community, uses every day. And I think sometimes folks don’t realize that these tech fundamentals came out of work to make products accessible for blind and low vision people. There’s tons of examples like this from the OXO company that makes household goods, whose original inspiration was to make household goods that were accessible for people with motor difficulties. Turns out, that just means good products for everyone. There’s so many more examples. The idea I’m trying to hammer here is that universal design and accessible design isn’t just good karma, it’s good business.

This concept is sometimes called a curb cut. Pictured here, I have a literal cut in a curb. And the reason that analogy is used is that curb cuts, while fairly ubiquitous in the United States, are a thing because of legislation designed to protect the rights of people with disabilities, people using wheelchairs. But anyone who’s ever used a stroller or luggage or a shopping cart can attest that these are a profound net positive to our cities and communities for everyone, not just people who use wheelchairs. And the same concept applies to so many other things. The bumps on the F and the J keys on a keyboard. Subtitles. I myself don’t have hearing loss, but I, I turn subtitles on all the time. If I’m trying to watch a Christopher Nolan movie and understand the dialogue of what’s going on without waking up the neighbors, it’s sort of a necessity.

And the other side of that coin, descriptive video, which I often have running when I’m watching movies with mom, actually has often helped me understand the plot of the movie when it’s maybe going over my head a bit. So I think that there’s an opportunity to build products that aren’t just good for the blind and low vision community, but use that aspect to make good products for everyone.

Pictured here, I have a, a shot of Sophia Gallo. She’s one of my colleagues in Seattle. She’s blind. She’s a UX researcher, and she works with me and a few other members of our team to build up early hackathon projects. For those who don’t know, hackathons are work that people do at a company that’s not strictly part of the day job. Often, they take place over a weekend or a few days. The idea is to come up with cool ideas that your boss didn’t tell you to do, build them out, and see if something great comes out of it. They’re a big part of the history of major firms in Silicon Valley, including Facebook, now Meta. And many of the great things that have come out of the company I work at, like Messenger and other things like that, actually started as hackathon projects that weren’t anyone’s assignment.

I bring this example up because while multimodal AI and the AI features of our smart glasses are very much talked about now, this hackathon project took place in 2022. Uh, the RayBan Stories, the previous version. It took place before the AI hype kicked in in ’23, and it was the first implementation of AI on our smart glasses products. When we showed it to our leadership, many of them were won over immediately. And while I certainly can’t take credit for making AI part of the smart glasses roadmap, this was certainly a thing that was bound to happen, I do feel immensely proud that…… the origin story, the first implementation of AI in smart glasses was actually an internal project to help bring the world a little bit closer to blind and low vision people.

In this case, we took much of the feature set of the wonderful Seeing AI, uh, implementation that Microsoft have built on mobile phones, and we took many of those features and made them possible to use through the camera on the smart glasses. Now, a lot of things have changed and moved since then, and the implementation that we do on Ray-Ban Meta is much more advanced than that first hackathon project was. But the principle remains the same. A really good idea to support an important community that is sometimes not seen enough by mainstream tech helps make a mainstream tech product better for everyone.

Now, this took off more than, uh, even I expected, and, and much more than our leadership expected. I’ve got an image here of Mark Zuckerberg on stage at Connect, our big, uh, yearly conference on technology from last year unveiling our partnership with Be My Eyes. Just the concept of our tech CEO talking directly about the importance of accessibility brought tears to my eyes. Because up till now, most of the places I’ve worked, I’d worked on really cool accessible ideas as a side part of my day job, and it was always seen as a nice thing that was cool, but, “Well, get back to your day job.” And here, I saw the output of a lot of that passion work go into real material improvements in our products, um, that are helping people every day. Which I think is awesome.

It’s something special about Meta as a company, the culture that lets good ideas come from everywhere. But I think it’s also something really special about the technology we’re working with, where implicitly, if we build things that are good for the blind and low vision community, the products we build are just good in general.

Now, like I said, this took off faster than I could have expected. I’ve got a shot here of a headline. I remember my brother calling me up, um, about mid last year telling me, “Hey, your work’s in the Wall Street Journal, Gus. Isn’t that cool?” And I, I freaked out for a sec because usually if Meta’s in the Wall Street Journal, it’s not always a good thing. (laughs) But in this case, this headline reads, “Meta’s AI-Powered Ray-Bans Are Life-Enhancing for the Blind.” And this is just one of many examples that these were not puff pieces that our comms team put out. These were organic things that actually surprised much of our own leadership, ’cause they didn’t realize how much this product would take off within the community.

Um, and so while I feel immensely proud that we built these, I also think there’s a long way to go to actually let the wider tech community acknowledge how important this work is. We do have an amazing portfolio of things coming up. I’ve got a shot here of our Oakley Voyager glasses, which are sporty wraparounds. They’re waterproof and designed for high intensity engagement. A shot of some of our next gen Ray-Ban Meta Glasses, and a picture of our Oakley Houston stylish frames. I love the idea that we can build this technology into styles that’ll appeal to everyone. And I also love the idea that we’re not getting stuck in the idea that technology for people who are blind has to look like something that’s for people who are blind.

Of course, I think that it’s great to have pride in who you are, and as a person with some vision loss living with these extremely thick glasses, which those of you who can see can tell on the camera, do set me apart a little bit. I don’t see any shame in, in, y- being who I am. I think it’s great to be, take pride in being low vision and being blind. But when it comes to buying technology, we are people first, and we like to just buy stuff that’s cool. And when it comes to glasses, like, I think it’s great that the types of features we’re talking about are being offered in styles that are just appealing in general, they aren’t a niche thing that’s just for the community. It’s great from the standpoint of design, and it’s also great from the standpoint of economics.

When you look at some of the other devices out there that are sold for thousands plus dollars, those price points aren’t because the developers are greedy. Far from it. The people who make other products that are similar to the feature sets are amazing people, and much of the work that is being shipped now is built on the shoulders of those giants. But economies of scale are a big part of consumer technology. I might be spending hundreds of thousands of dollars plus on an injection molding tool to make a part, and that costs hundreds of thousands of dollars whether I make 10 parts or a million. So, if I can’t make a million and sell a million, guess where that price goes?

So, I’m really excited. I think we have some amazing stuff going forward, and there’s some really cool technology that’s being worked on both at Meta as well as elsewhere, but I think the best stuff is being worked on here, that I think is gonna empower other communities with disabilities. We also announced and released our Meta display glasses, which I think will provide really cool opportunities for people who have light perception but maybe poor vision, as well as really great feature sets for people who are deaf and hard of hearing, like live captions. And just like my example before about curb cuts, something like live captions is useful not just for a deaf person, but when I take that live caption ability and I put in translation, now I’ve just made a live translator. And you can also do audio-to-audio translation. So, if I’m wearing audio-only glasses like Ray-Ban Metas, I can still translate a conversation that I’m having with a friend who might speak Spanish while I speak English.

I’ve also got pictured here our Neural Band, a really novel input device that can tell what signals you’re trying to send your muscles in your hand without needing physical input from your hand. The kind of thing this can do, today we’ve shown it as a cool way of moving and interacting with a product without needing to hold a controller or keyboard, and doing it discreetly. So for example, if I wanna ask my glasses to describe what’s around me or tell me if the person that I thought I was talking to is still there, I might be able to do that with a hand gesture instead of saying, “Hey, Meta,” and it being a little bit awkward.

Um, but I think it’s also profoundly impactful for people who might have motor difficulties. Um, we have examples internally and with close partners of folks using this technology on a arm that has lost physical motion, but the nerve fibers are still there and they can still control things as though they still had a hand. Um, it’s just really amazing stuff and I’m, I’m super excited about where the future leads.

I’ve got another shot here of the inside of our display glasses, showing what it looks like, and the concept of potentially using this as a way to magnify or make text more legible or make contrast cues more visible for somebody who has low vision.

So, I go back to all these stories for a few reasons. One is, this is only the beginning. The things I mentioned, the capabilities that I touched on are available today, but there’s limitations. Um, I still think that a human assistant is often more reliable for certain tasks, and I’m, I’m thankful that we have cool things like Be My Eyes and Aira as possible avenues to get help remotely from somebody when we need it. But I think we’re also getting to a point where a lot of things that required stepping away from our independence are gonna be unlocked.

Closed captioning, for example, for the deaf and hard of hearing community is already fairly well-automated by AI. It’s not perfect, but 90-ish percentile quality. And we’re getting close to a point where you’re gonna start seeing that with video content. Uh, descriptive video is woefully, uh, unavailable on many great pieces of content, um, ’cause it’s labor-intensive. It’s not ’cause people don’t care, but it, it does require investment and work. I think that reducing the bar and the lift to doing that with AI is gonna be just amazing for so many people, and the natural next step to that is live description of the actual world, which is super exciting. We can put our toe in the water on that today with Ray-Ban Meta glasses and Oakley Meta Glasses, um, but it’s just gonna get so, so much better. And that’s just one example of what AI can do to empower people in the community. There’s many, many more.

So, one quote that me and a few of my colleagues learned over the past couple years is, “Nothing about us without us.” I think it’s super important. Like I mentioned, a lot of that Hackathon project work that I touched on with, uh, our first AI on Smart Glasses Hackathon, that happened because of people like Sophia who actually are blind and have lived experience, and people who can directly inform product because they face the challenges every day.

Um, I think there’s a big difference between building with a group and building for a group, and while the intent is often really good for people to try to build for a community, I think it’s really important to break down that barrier of us, them, or otherness. Um, I think the best thing that tech companies can do to really make things good for everyone is to hire people from communities like the blind and low vision community, to consult with them, to involve them in the design process. And conversely, I know it can be really tough to kind of break through the barriers that exist to get access to opportunities, but the more we in the blind and low vision community involve ourselves in product design, the more we can help shape things to be better, not just for our own community, but actually to make products better for everyone.

So, I want to maybe close off this talk reflecting on what other ways technology can empower us to be independent, to connect to the world, which Meta’s mission is to connect the world, and to lead fulfilling lives. Thanks very much. I’m happy to chat more about this, and, Karae, I look forward to our conversation after this. Thanks.

KARAE LISLE: Well, that was really a wonderful presentation. I just love the way you told the story. I think it really sets a great foundation for how things are moving so quickly now, right? You were able to cover 40 years in three minutes, and that’s about what we’re doing (laughs) in real time with AI right now. So, um, I do have, uh, some questions that I wanted to ask with regard to, um, just explaining a little bit more about the glasses, uh, some of the features. So I know we’ve got, um, uh, it can do sensing. It can give us audio and visual feedback. There’s haptics. Um, can you just kinda show us the glasses and, and give us some maybe use cases of, of how, uh, people will be using them or they are using them now?

AGUSTYA MEHTA: Sure. So, it’s funny, ’cause, like, what I found with these glasses when I’ve gone to different conferences, often our cross-functional partners or groups I haven’t even heard of are selling them on our behalf, which is, like, amazing. That’s, like, a great problem to have. Um, so there’s use cases for these that, you know, even I hadn’t, hadn’t initially considered as somebody who works on the team, but I’ll, I’ll walk through a little bit.

So here, I’ve got a set of our Oakley Voyagers. This is one of our newest form factors of the glasses. To describe it a bit, it’s the classic wraparound sunglasses look. I’m gonna put ’em on right now over my normal glasses. Um, I’m not sure I can quite pull it off, but you know, maybe if I, maybe if I gained a few pounds of muscle or something. (laughs) Now, they’re, they’ve got a shiny sunglasses type visor. They curve around your head. There’s two arms on the side that keep it on by wrapping around the sides of your head, and inside of these arms, there’s a, a lot of really cool technology embedded.

There’s a world-class mobile processor that can run some really cool things on the device itself. There are some amazing speakers that provide great audio that couples mostly to your ear and not to the outside world, which I think is key, because it means that your stuff is pretty private. You crank the volume all the way up, some leaks out, but in general, normal volumes, it’s, it’s a personal, private experience. But it doesn’t cover your ears up, which I think is critical. Um, you know, I, I love using earbuds and over the ear headphones when I’m jamming out to music, but it kind of closes you off from the world. And something I think a lot of folks don’t realize is how much spatial audio cues from the real world help us orient what we’re doing, whether sighted or not. And for people who are blind and low vision, it’s especially important not to cover the ears up. Um, but, but even if you have vision, losing that sense, it, it’s an amazing thing to see how much it takes away from your balance and your ability to avoid obstacles.

Now, um, we’ve got a world-class camera here as well. In the case of the Oakley Voyagers, 3K resolution centered on your face, um, which is a bit different than our previous generation of Ray-Bans where it’s off to the side. It provides a first-person view of what you’re doing, which Ray-Ban does as well, but I, I love that the camera’s centered on this product so it’s really, like, spot in front of you. You don’t have to worry about whether to move something to the left or right. Um, this particular version is waterproof, it’s rugged, um, it’s really fun. Um, very sporty.

Now, in terms of the feature sets and the things it can do, I think first and foremost, it’s like a really great personal audio device. Um, I found just using it to listen to music is really great. Quality’s quite good. Using it for voiceover is a really fun experience because of the, the things I mentioned around privacy and discretion without covering your ears up. Um, and it, it’s connected by Bluetooth to your phone. The phone’s doing a lot of the heavy lifting with the Meta AI app. Um, and so you essentially have hands-free access to much of the content on your phone with these even before I get into the unique Meta AI features.

Now, I think the special AI stuff that, that this stuff, that these, this product can drive is, is where it really becomes unique. Um, you can ask your glasses in a natural voice to read you a restaurant menu, to describe what’s around you, to tell how many people are around you, um, to understand whether the room you’re in has the lights on or not, to tell you what kind of currency is in your hand, um, and many, many more things. Is it perfect? No. Uh, like all AI, I always have to put the disclaimer that AI doesn’t always give 100% accurate answers. But is it a life changer and a game changer for how so many of us operated before, though? Certainly.

I mean, our pantry back at home, my dad had spent quite a bit of time developing a QR code system where he taped codes to different household goods in our house so that my mom could use the scanner on her phone. And now that whole thing is obsolete because she can just ask whether the thing she’s holding is a can of soup or a can of pumpkin. Um, I bring that example up because that literally happened a couple of days ago in the process of making some pumpkin pie that thankfully was not chicken soup pie. (laughs) Though I like chicken pot pie, they’re different things. (laughs)

Um, so I think those are some of the quick features. Some of the new ones that have come more recently and I think are really exciting are, um, live AI. So the ability to not have to queue up the assistant with, “Hey Meta,” every time, but to just set it into live mode where you can talk naturally and ask it questions. That’s been really exciting. Um, great feedback on that.

We recently released a feature called detailed descriptions, which you can find under our accessibility subsection of our settings. Um, and the premise of that is to provide more verbose, detailed output when you ask it a question because sometimes all you need is a summary, like there’s a person with a dog. But sometimes all that really cool detail and color about what kind of dog it is and what it’s doing is something that we miss out on and we can’t see, and it’s great to have access to that again.

Um, other great things that are part of the features that we’re looking at, we also have really amazing, uh, remote assistance features. I mentioned Be My Eyes is one of our close partners in this venture, and, um, we’ve seen really great response and uptake from the community on using Be My Eyes in conjunction with smart glasses because it’s just much more natural to have the camera view first-person view. Um, so that’s maybe a quick summary. There’s much, much more.

KARAE LISLE: Well, that’s really helpful. And o- one of the things I really like about the way you’re describing it aligns with how we provide instruction and how we serve our clients, which is first you want to be safe, right? Which is obstacle detection and, you know, where is the street, uh, where is the door, um, who’s around me? And then you want to become independent and or stay independent. And then once you’ve got those kind of basic things, right, which is human nature, then you want joy. And you talked about that you can listen to music or you can get a detailed description if you went to a museum or a park. And it’s not just a tree, but it’s a tree that is experiencing the fall and the leaves are turning orange and, and, and brown or, um, yellow. And I think that’s where we really start to see equity in the world, right, for is, is, is the experience, we’re experiencing the same things. We know about the colors if we or how we understand colors. And so I really love that, that this technology is aligning with, with humans and the way humans speak and feel and live, right? Um, and that, uh, that’s just such a wonderful part of community. So I really love the way you described that.

And I was wondering, um, as well, uh, when you, when you talked about, um, the, the design decisions that you were making and things and how important it is, um, to have users, uh, um, uh, uh, testing and giving feedback, can you share a little bit more about that experience and some of the recent, uh, upgrades that have come out?

AGUSTYA MEHTA: Yeah, certainly. Um, I’ll be the first to admit as well that I think a lot of the amazing success we’ve had with, with Ray-Ban Meta I think was serendipitous. I think in many ways, the design decisions that make a product good as a pair of glasses that don’t have a display are implicitly good for the blind and low vision community. There was certainly a lot of stuff that went in through work like the hackathon I mentioned and through people like myself and colleagues of mine like Sophia. But I think we also kind of fell into a good place where making a good set of smart glasses, like, implicitly needs to be accessible.

Since then, before then to some extent, but since then, even more so, I’ve seen a lot of really great examples of getting feedback from the community and it helping shape our design decisions. Uh, a couple good examples I will give, we spent a bunch of time with leaders from NFB, ACB, um, a lot of back and forth discussion, uh, co-design activities with members of the blind and low vision community, where we had them beta test new features. And we brought developers in to kind of hack new ideas together.

And what we found is, like, some of the assumptions that were being made around the implementation of multimodal AI were, if you hold a piece of paper, you want it to just be summarized. This is a letter with a bill on it. Maybe that’s what you want sometimes, but a lot of times, I literally want you to read the bill out with all the numbers. Um, and so that toggle that now exists with detailed descriptions, detailed responses, is, is an output of some of that engagement we’ve had with the community.

Um, I think there’s a few more examples of things like that. I also will be the first to admit that there’s a long way to go. Um, I want to use this platform as an opportunity to engage with the community, get ideas, um, but I am always very cautious to avoid, you know, trying to take a victory lap or gloat that, like, everything my company does is awesome and amazing in the space of accessibility, because I know there’s a lot r- for improvement. And so a big part of this is actually trying to get more feedback as well.

KARAE LISLE: Mm-hmm. I think that’s important. And I know that, um, we really appreciate the opportunity to do user testing for, uh, Facebook and Meta, um, and we… It’s a great opportunity for our clients to say, “Yeah, here I am getting to, to provide user feedback to, to this very large company that’s creating, you know, um, uh, ubiquitous technology and they’re asking for my opinion,” right? And so even sometimes when we’ve done the user testing with, with the larger companies, uh, we get specifications on what type of, we want some blind users, we want some users with this kind of, um, vision, peripheral vision or tunnel vision, to really, really start to understand all the different elements of what does it mean to, to use this tool, to use this device as a blind or low vision person.

And so that, that kind of, um, definition and that understanding that technologists are starting to ask for is really encouraging. It’s encouraging to companies and nonprofits like Vista Center and to our clients. They get so inspired that they’re being asked, right? And that they’re listened to. And, and we can really see that in, in this generation of technology that’s coming out with AI. So I really, I, I really, uh, wanna, uh, commend you, and I, I do think there’s a small victory lap there for sure. (laughs)

And then, um, let’s talk a little bit more about some of the, the hardware as far as the, um, the battery life, um, the, um, the other kind of mundane elements of having wearables, because people do ask about that, you know? How hard is it to actually keep them charged or, you know, do I have to put them in a special case? How do I carry them?

AGUSTYA MEHTA: Yep. Um, it’s funny, ’cause I, I think the, the way you describe it, um, I think most of the c- the, the, the world thinks of this stuff as ma- mundane. The bread and butter of an engineer is often, like, spending hours on these things that might seem trivial but they’re actually very, very hard and complex.

KARAE LISLE: Very important.

AGUSTYA MEHTA: Yeah, exactly. And, uh, you know, the one you touched on, battery life, is a key one. Now there’s, there’s actually a pretty big improvement in battery life between the previous generation of, uh, Ray-Ban Meta glasses and the updated Oakleys, as well as the updated Ray-Bans. Um, so we are making improvements and strides in that direction.

But fundamentally, there’s always trade-offs and conflicts in engineering, and in this case, the trade-off is we want a device that’s small, sleek, nice-looking, doesn’t weigh a ton, feels good, looks good, um, which demands not much physical volume, and then having a large battery means you need a lot of space to put it. There’s also a big trade-off on what we call thermals. Basically, a lot of the components doing the magic of AI and the process thereof make a lot of heat, and so you need space, you need engineering solutions to get that heat out of the product. Um, you also need power to power that stuff, which ties into the battery capacity. So that, that trade-off on power h- hits you twice, right? You need to have a physically big enough battery to power your stuff, and you also need enough physical space on the device to make sure that you can get the heat out of it.

Um, now, we’re gonna see more and more capability coming out of smaller and smaller devices. That’s the progress that’s being made in the world of silicon, right? If you look at what a small mobile chip that’s in a modern mobile phone is capable of in terms of compute power, these things outclass… You know, the analogy that’s often given is the computer that took us to the moon, but I’ll even go beyond that, right? I mean, the, the type of thing that would have required a really expensive super computer or years later, a video game console to render, you can, you can do on a mobile phone, right? I mean… In fact, a lot of people don’t realize, but like, the Nintendo Switch, which is, like, considered a game console that, like, is in the same rank as things like the Xbox, the type of chip they use is the same sort of silicon that’s used in mobile devices like phones and stuff. It’s not like a, you know, there’s not much difference anymore.

So power draw on silicon is gonna get better and better. Capacity of silicon to do things, particularly tasks associated with AI compute is gonna get better and better. Um, I think that we’re gonna get to a future pretty soon where you won’t need to, you know, constantly charge the device with the case. I think we are a bit of a ways off from having the full capability of a mobile phone in the form factor of glasses, but it will happen. And I think we’re also going to get to a point where a lot of the tasks that people want to do won’t require a network connection. Today, most of the AI lift is taking place outside of the device. There are some things that can be done on the glasses, like the OCR, active-taking texts, uh, from an image, that can be done here, but most of the L and magic that describes what’s around you requires being connected to the internet. Um, but we’re, we’re already seeing models in, i- in, in, uh, implemented that can fit on mobile devices, that can do some pretty cool stuff, and I think you’re gonna start seeing some of that percolate to wearables, um, so that there’s some capabilities that don’t require you to use others.

KARAE LISLE: Yeah. Good, good, good. And I had a couple of questions specifically from our users. So I want to thank some of the folks at Vista Center who provided me this. Um, so, uh, the understanding that we have is that Meta is developing their own mapping system rather than relying on Apple or Google Maps. Could you share how Meta’s maps will specifically enhance mobility and independence for people who are blind and low vision?

AGUSTYA MEHTA: Sure. So one thing I’ll, I’ll preface or disclaim is, like, in terms of things that are coming out with the future and things that aren’t released, I, there’s things that I can’t talk about certainly, so I’m not gonna speculate on, like, what our future plans are for first-party mapping or what they aren’t. But, um, I think there’s actually even deeper part of that question which I want to address, which is, well, is this stuff not gonna work with the things that I already use? Um, I think the short answer to that is, like … take the example of the iPhone, right? Like, Apple has done a lot of really cool stuff with it, but they didn’t do it all themselves. I think one of the things they did, which didn’t come out when the first iPhone came out, came out later, was the App Store. That’s one of the things that caused that to take off as a platform.

Um, analogously, you know, we just announced and released our developer access toolkit. Uh, essentially it’s a mechanism by which developers can implement their ideas using our technology. Um, and I draw that analogy because there’s a lot of first-party stuff that we’re gonna do, and I think it’s gonna be awesome because engineers at Meta are great and we make good stuff. But not everything needs to be done by us, and I think that what’s really gonna make this platform an amazing one is partnership with developers across the community. So I’ll kind of counter that question with, if there’s a mapping implementation that you find really good, work with the developer or, if you’re our developer, build it yourself and partner with us using, uh, our developer access toolkit so that it works on our device. Um, I find that to be a much better approach than, like, hoping that one of the big companies comes out with something that ends up being good.

KARAE LISLE: Yeah. I like that answer. I think that, uh, that was a point that I had had as well, was, you know, how, how big will the ecosystem be? Um, so we’re getting, uh, close to the time of our, the end of our session, and I wanted to, um, give you, uh, just an opportunity to kind of summarize. You know, you’re talking to thousands and thousands of people in 70 countries, and, um, how are you feeling about accessibility, uh, in the world, and, and, you know, you’re a designer and you’re a user, um, and you’re the son of someone who, uh, uses technology. So close it out for us.

AGUSTYA MEHTA: I’m feeling optimistic, and this is coming from someone who tends to be a skeptic. Um, I, I grew up hearing people talk about some amazing thing they read about that will cure every problem anyone’s ever had, and I think you all probably experienced stuff like that as well, and then it gets you kind of jaded at a certain point. But seeing mom use this technology and seeing how it’s enabled her, dabbling in it myself, and watching my sons, I have three boys at home, my youngest inherited, uh, night glaucoma and congenital cataracts. You know, it’s easy to get into a, a spiral of despair about these things sometimes by overly focusing on the negatives.

I just feel so humbled and honored to be in a position to influence the world of technology and to be a part of the story of technology at such a cool inflection point. To me, I think the advancement AI is gonna bring to the world is gonna be on par now just with the advance of the internet, but potentially, like, the Industrial Revolution. I know that’s, like, a really big thing to say, but I think it’s going to be something like that. And just like the Industrial Revolution brought so many people out of poverty, I think this type of technology is gonna bring tons of people who society has previously left behind or missed and bring them to the table. So I’m optimistic.

KARAE LISLE: I’m glad to hear it. And Sight Tech Global is exactly the platform to make those kinds of comments. So I really appreciate you ending on such a positive note. I certainly share in your optimism, and, um, Vista Center wants to just, uh, match your strength and your optimism in serving the clients that you’re building the tools for. So thank you, Gus, so much for, um, what you’re doing on a daily basis and for partnering, uh, with, with Sight Tech Global and all the other smaller, um, and enterprise companies that you all are, are partnering with. We really appreciate you and you’re making a big impact. Thanks for joining us today.

AGUSTYA MEHTA: Thank you. Thanks for having me.

[MUSIC PLAYING]