Fable: The barriers to utopia: Why feedback comes first

A lot of conversations these days are about the latest technology, and how it promises to solve all of our problems. But what about people? Join the CEO and the Community Lead of Fable, Alwar Pillai and Samuel Proulx, as they discuss how to collect authentic feedback from people living with disabilities.

• Alwar Pillai, CEO, Fable
• Samuel Proulx, Community Lead, Fable

ROBERT FRAWLEY: Well, hi, everybody, and welcome to our breakout session for today. Today’s title is the barriers to utopia: why feedback comes first. My name is Robert Frawley, and on behalf of Site Tech Global, I am so excited to have you join us. In today’s 30 minute breakout session hosted by Fable Tech Labs, you’ll hear from Alwar Pillai, CEO at Fable, and Samuel Proulx, Community Lead at Fable.

Before we begin, just a couple of housekeeping items. This session is being recorded and will be available post event on demand on our YouTube channel and will also be added to our agenda. If you have a question or comment, please use the Q&A box that you’ll find in your Zoom browser. And with that, please welcome Alwar.

ALWAR PILLAI: Thank you, Robert. Sam and I are really excited to be here today to talk to you about why feedback comes first. Before we get into the topic, we’d like to give you a little more, learn more about us. So Sam, why don’t you kick it off?

SAMUEL PROULX: I’d love to. Super excited to be here. Hi, everyone. I’m Sam Proulx. I am the community manager here at Fable Tech Labs.

So my day job involves recruiting, supporting, and helping out the excellent and diverse community of folks. All of whom live with a disability and use assistive technology that we work with here at Fable help test our customers products. I myself am a screen reader user and have been using screen readers for over 30 years across all kinds of different platforms. So accessibility is something that I’m extremely passionate about and that is very close to my heart.

And that’s why this talk is so important to me today. Because, if the research that we’re doing doesn’t include people with disabilities, how will the product that we wind up building ever be accessible, and be usable, and be a good experience for people with disabilities, unless we’ve done the research to make it so? And if the product isn’t accessible and isn’t a good experience, that, of course, can have a massive impact on people with disabilities, especially in today’s online world, where you may be unable to apply, or get jobs, or just participate in daily life with the independence that everyone with a disability needs and I believe is entitled to.

ALWAR PILLAI: Perfect, and hi, everyone. My name is Alwar. I am the co-founder and CEO of Fable. My background has been in design. I’ve been also an accessibility specialist, an accessibility manager in organizations, and see the way companies approach accessibility.

It’s always focused on remediation. It’s focused on compliance, and the big thing that was missing was the voices and perspectives of people with disabilities. And that’s the reason we started Fable was to help make it easier for companies to engage with people with disabilities throughout the research and development process.

The reason this talk is important to me is because two things. There is the way we build products has changed, and it constantly changes. And the way users engage with products also constantly change. We have new technologies. There is new interactions, but unfortunately, our research and research methods haven’t transformed a lot.

And we think that’s what’s needed to start integrating the perspectives of people with disabilities. So today, what we’re going to be doing is we’re going to be talking about three aspects. One is why current research processes are limited, and it’s making it harder for us to collect feedback from people with disabilities.

The second is what are the basic methods to collect authentic feedback, and the last part is once you’ve collected feedback, what do you do with it? So let’s get into why current research processes are limited, and how we can engage people with disabilities. The first one is think out loud.

Think out loud is an extremely common method that organizations use to identify how usable a product is. It’s simply asking a user to go through a product or go through a task, and ask them to verbalize their thoughts. This has existed over 20 years.

It’s a really cheap method and robust process to collect feedback. All it requires is a participant to perform a task and to talk about their experience. Sam, what do you think are the challenges with this process when it comes to people with disabilities?

SAMUEL PROULX: Well, you know, as a screen reader myself, I’m sure there are probably one or more screen reader users in the audience who immediately thought, hey, this might not work well for me. Because, if I have to talk and explain my feedback at the same time as I’m performing the task, I have to listen to two voices at once. I have to say something intelligible, and I have to be listening to my screen reader.

And, of course, even if you could talk intelligently while listening to your screen reader and performing a task, it would wind up being a problem for the researcher, who now has to listen to and comprehend two voices at once. One of them being an unusual robotic screen reader voice, so it can even be difficult for them to understand the feedback from think out loud without some care. As well, it can be a problem for users of Dragon Naturally Speaking or other voice recognition software.

The primary way that they perform tasks on their computer is by talking to the computer. So obviously, if they’re talking to the computer, they cannot be thinking out loud and talking to the researcher at the same time as they’re performing the task and as they’re interacting with the computer. Because the computer is likely to interpret some of those voice commands as instructions, or feedback, or things to be dictated when that wasn’t what was meant at all.

ALWAR PILLAI: Definitely. The second is not a process or a method. It’s more a structure and format, which is usability labs. So organizations often have a usability lab in their premise. It’s usually two rooms. One room, where a researcher and the participant is engaging, and then you have an observer room, where many people from the design team or digital team can observe the interaction.

The purpose is to evaluate the usability of a product. This is a method that is convenient for research teams, because all they have to do is invite a participant in to their location. But there are a lot of challenges when it comes to people with disabilities being able to go to a usability lab. Sam, what are your thoughts here?

SAMUEL PROULX: Yeah, unfortunately, while it’s convenient for the researcher, it is not necessarily very convenient for the participant. And that can be for a number of reasons. First of all, when you’re at one of these usability labs, you may wind up using unfamiliar technology.

It’s very unlikely that the usability lab has exactly the same version of your screen reader or other assistive technology installed that you do. It’s unlikely that they are running sort of the exact same operating system. And lastly, and most importantly, assistive technology is something that everyone configures and customizes to fit their own needs. And it’s certain that the version installed in the usability lab is not going to be configured the way the participant likes it and the way the participant is used to using it.

Secondly, of course, you’re in an unfamiliar environment, so the environment may be distracting, may not be comfortable. And there may even sometimes be accessibility issues either in accessing the environment or the environment itself. And when we talk about accessibility, that just naturally leads into travel.

Travel can be extremely challenging for people with disabilities, especially who don’t drive. Public transit is not always as accessible as we would like it to be. Paratransit is not always as fast as we would like it to be. So what ends up happening is that often, the travel to and from the location can take up way more time than the actual study does.

And lastly, it limits the ability for more users to participate. Because now, you’re only participating. You’re only doing research with users who are local to you. You’re only doing research with participants who can manage to travel to your location, and you’re only doing research with participants who are willing to go through all of that discomfort, and irritation, and extra time. And frankly, unfortunately, some people just aren’t willing to go through all of that, and I don’t really blame them.

ALWAR PILLAI: Yeah, I think that’s completely understandable. And while, sometimes, methods can be convenient for research teams, it’s important for us to take into the considerations of the convenience of the participant. And if you do have a participant with a disability who has gone through all of this to make it happen, how much burden have you put on them?

And what is the impact of their engagement in that current situation? So that’s something for us to keep in mind. The next one here we have is the SUS survey, so that’s the system useability scale. It’s a quick and dirty reliable tool for measuring the usability of a product. It consists of 10 item questionnaires with the fire response option, so it goes to strongly agree to strongly disagree.

This is an extremely convenient way to collect feedback. Because all you need is a survey that you can distribute easily, whether that’s in person or even online. And it can be applied to really small sample sizes or even large sizes, almost like thousands of participants. This one also has some limitation when it comes to collecting feedback from people with disabilities.

SAMUEL PROULX: Yeah, the primary limitation that you encounter there is that the system usuability scale assumes that all user needs are the same. It doesn’t consider the usability needs of diverse users. And it doesn’t take into consideration any of the needs of users who are using assistive technologies, or anything about them, or the challenges that they might encounter.

It was interesting to me when I was doing research into the kind of SUS method that when it was published, the original author said that you should customize this to your individual needs. And yet, it doesn’t seem to have been customized a lot. I know those are sort of some very high level problems with SUS. So I’d like to take a minute to kind of go over a couple of the questions that it asks, and why those can be a bit of a problem.

One question that the system usability scale asks is, I think, that I would need the support of a technical person to be able to use this system. Now, if you’re a screen reader user and the website just isn’t working properly with your screen reader, you don’t need the support of a technical person to use the website. You just need the support of someone who isn’t using a screen reader, so that question really doesn’t say much about your experience.

Similarly, I imagine another question it asks I imagine that most people would learn to use this system very quickly. Now, if you’re a person with a disability using assistive technology, that question can be a little bit confusing. What does most people mean?

Do they mean most people using my assistive technology? Do they mean most people with my disability, or do they mean most people as in are they asking me to imagine what it would be like for someone without a disability to use the system? And, of course, as any researcher knows, when you ask confused questions that a participant is unsure how to answer, you don’t get great feedback.

ALWAR PILLAI: Definitely, so we walk through three methods, formats that are biased against people with disabilities and their ability to provide feedback. On the screen right now, we have various other methods that are used right now in research and to identify how usable a product is, whether that’s card sorting, first click testing, eye tracking, five second test. Each of these have some level of bias, and as researchers, we should recognize that they have bias.

And the decisions we make will influence who uses our product. That said, we do understand that no one methodology will work for everyone. Very similar to design and how you say one size fits all versus one size fits one, I think that still applies, even in research.

So what we’re suggesting is that research methods should change based on the participant. And while this might be a little inconvenient for research teams, because it’s always easier to have one method, collect a ton of data, and then work on it, we think to make sure you’re actually having diverse perspectives, you need to use different methods simultaneously to collect feedback. So what we’d like to move on to now is what are the methods to collect authentic feedback, and at Fable, we started two and a half years ago.

We’ve run thousands of engagements with people with disabilities, people who use assistive technologies. We ran it in different formats, moderated, unmoderated, and we have learned a lot. So for this section, we want to provide some quick tips that we think would help you get started on how to engage and collect authentic feedback.

The first one here we have is continuous engagements. So it’s important that you continuously and regularly engage with people with disabilities throughout your research, design, and development process. Often, when it comes to accessibility, companies think of it as a one time fix. So they will have a dedicated accessibility project, where they think that we can tackle all of these accessibility issues, and then our product will be accessible.

That doesn’t work, because our products change. Our OSS updates. Our browsers update. Our devices update, and the technology that people with disabilities use also constantly update. So for you to know how accessible and usable your product is, you need to regularly engage and test. The second on what we’ve learned is that more empathy leads to better design.

And we’ve seen this at Fable, where we’ve had some developers take part in moderated sessions and watch a screen reader user use a product that they built. And the screen reader user is unable to perform the most simplest functions. This really hits home for people of the fact that they build something that actually excluded someone, and we think that the more engagements that digital teams have with diverse people is how the needs of diverse people, it’s going to be more top of mind.

And then the last part here is that an inaccessible design cannot be retrofitted to become accessible. Companies often think about accessibility later. So they build a full product, and then they’re like, now, how do we make this accessible? And when you do that, you’ve not thought about the user experience of a person with a disability from the get go. So now, you’re trying to do patchwork solutions to make the product “just being able to access.”

So it might take a screen reader user like 15 clicks to get to something that another user would be able to do in one click. Is that experience accessible? Yes, they are able to access it. But is that a good user experience? No. So we think by having more regular, continuous engagements, you can actually build a more inclusive product. Sam, why don’t you talk about remote testing?

SAMUEL PROULX: The second method that we have here is remote testing. And as we talked about a little while ago, usability labs and in-person testing can be very difficult for people with disabilities. And we found at Fable that remote testing is an excellent solution to this problem and is an incredibly great way to gather feedback from people with disabilities.

There’s a few reasons for that. First of all, it allows you a much wider reach of participants. Instead of being restricted to participants who are just local to you and where you work, you can expand, and you can test with participants from across the country or even around the world. It also makes scheduling a lot easier. Because now, there is no longer the burden on the participant that they need to book their travel three days in advance, and they need to sort out how they’re going to get there and how it’s going to be accessible to them.

Instead, they’re joining online from their computer in an environment that’s already comfortable for them, and that also makes it fast and convenient. Because now, you are no longer kind of having a usability lab, and you no longer have to think about how to design that in person space, and how to provide the technology, and how to configure it, and all of those things. Instead, you’re working with a participant using their technology that they’ve configured, and all of that means that you’re testing much closer to the real world.

And that’s super important to help you understand how people with disabilities will actually use the product and what their experience will actually be like. Because now, the feedback you’re getting is from someone who is in an environment that is comfortable for them, who is using equipment that is familiar to them, and running accessibility technology that is configured and installed exactly the way they like it and exactly how they really would use it in the real world. And that sort of leads us into– as we talk about remote testing, it leads us into the importance of adaptability.

There’s been a lot of changes this year, and all of us have had to learn a lot about adaptability. But one thing that we’ve always known at Fable is that adaptability is extremely important and powerful when you’re collecting feedback from people with disabilities. And the first way that it’s important to be adaptable is in your format and in the medium of communication.

Because you want to hear from as wide a variety of different participants as possible, and there are some participants who will not be comfortable or will not be able to communicate in a video meeting. So for those participants, being able to collect feedback in a written form is very important, because it allows them to participate in a way that they could have never participated. And it allows them to have their voice heard.

Whereas similarly, there are some people who are just very uncomfortable with writing and would much rather participate in a video than a conversation. It’s also important to be skillset agnostic and to be adaptable in that way. Because, if you test your product with an expert screen reader user, something that is just an annoyance to them may be a complete barrier to an intermediate user or to someone who is just slightly less comfortable with technology.

So you can’t say that just because I’ve tested with one expert and it works for them that it’s going to work for everyone across the board. But part of being adaptable in that way is making sure that the platform that you’re using to conduct your research is fully accessible, and it’s easy enough to use that people who are not technology experts don’t find that they have barriers blocking them from actually participating in your research. And lastly, you need to think about contextual improvement.

One example of this is that companies often think about time to complete as a good metric for how well they’re doing, and how the experience is. Unfortunately, if you take, for example, the time a screen reader user uses to complete a task and compare it with the time a Dragon Naturally Speaking user takes and the time someone using a switch system or a head mouse might take, those times are completely and totally different. And comparing them doesn’t really mean anything.

Neither does averaging them, so it’s important to be contextual. One way that we find can be helpful is to test and retest with the same person. So if the first time it took them 45 minutes, and the second time, it takes them 20 minutes, you know that you’ve made some real improvement in how quickly they can complete the experience. But if you compare a screen reader user, a Dragon Naturally Speaking user, and a head mouse user, you’ve really learned nothing. Because those users are so completely different and have such completely different expectations as to how long an experience should take them.

ALWAR PILLAI: Definitely. I think the more adaptable you make your process, the more adaptable your product can be, and these are some good considerations to keep in mind. The last part here we’re going to move into is now that you’ve collected feedback, you have feedback from users. You have feedback from people with disabilities. What do you do with that data?

In an ideal world, feedback from users and users with disabilities will live together and be taking on action together. But in reality, the way we prioritize feedback is often based on like frequency and volume. And if we base it on these things, it automatically removes the outliers.

So what are the ways we can make sure that feedback from people with disabilities and feedback related to accessibility is prioritized? Some of the ways we have seen this work really well is by focusing on impact, and there are three questions here that we think is important to ask. One is, can users get access to the service? So are there multiple alternative ways?

Is there only one alternative way, or is there no alternative at all? And if that’s the case, then that becomes a high priority feedback to work on. The second one is can users perform critical tasks, so are they able to perform this completely independently? Are they able to perform parts of it independently, or are they not able to perform it independently at all?

And if that’s the case, then that becomes high priority. The last one was really interesting. Sam and I were chatting about based on what the product is. Because that needs to be taken into consideration as well. Sam, what are your thoughts here?

SAMUEL PROULX: Yeah, definitely. I mean, there are some products, where the action that you need to be able to perform is absolutely critical. If you can’t dial 9-1-1 when you need to, even if you maybe only ever need to call once in your whole life, that can be very critical.

However, in many applications, one good indicator of priority is how frequently will you perform a particular task. A really good example of this in my mind is when I’m doing my online banking. I order a checkbook maybe once every two years if that.

I just don’t write many checks, don’t use checks. So if I need to get sighted help in order to help me place an online order for a checkbook for my bank, well, I’m performing that task rarely, and I don’t mind so much having to get a little bit of help. But if it’s a task that I’m performing routinely, for example, logging in to check my balance, if that’s inaccessible and I need help to do that, well, I’m trying to do that once a month, once a week, maybe even more. So the impact that it’s having on my experience is enormous.

ALWAR PILLAI: So yeah, I think these are some questions that are good to ask. And I think each research team, as you’re developing your own metrics to help evaluate and prioritize, to think about what are the questions to ask specifically for assistive technology users to make sure that their feedback is considered. At Fable, like we mentioned at the start, our goal is to make it easy to collect feedback from people with disabilities.

When it comes to accessibility, there’s a lot of focus around compliance, around how to make sure you’re meeting the WCAG standards. But there’s not a lot of focus on how do you measure the usability for people with disabilities, and our customers have asked this. Everyone’s thinking about this, so we decided to give it a shot.

What is a way we can develop that it helps capture the usability of assistive technology users? So we’re excited to introduce the accessible usability skill, which we, in short, call AUS. It’s a quick and reliable tool for measuring usability for assistive technology users. The goal of this is to make it easier for organizations to broaden their research and include people with disabilities, but specifically assistive technology uses. This is similar to the system usability scale, but it’s different. Sam, do you want to walk us through some of the differences?

SAMUEL PROULX: Sure. So as we were creating the accessible useability scale, we really thought about all of those questions that I had talked about previously, and how they don’t necessarily serve people with disabilities. So we went through each of those questions and modified them to be easier and clearer for people with disabilities to answer and to take their feedback and experience into account. So to go back to that first question that I gave an example of as to why SUS or the system usability scale does not work, one of its questions was, I think, that I would need the support of a technical person to be able to use this system.

Well, we’ve changed that question. So it now reads, I think that I would need the support of another person to use all the features of this website, and that’s much closer to the kind of experiences that people using assistive technology will have and much easier for them to answer. Similarly, the question, I would imagine that most people would learn to use this system very quickly, has been changed to I would imagine that most people with my assistive technology would learn to use this website quickly.

And that really helps people give feedback, because now, they understand exactly what you’re asking them to imagine. And it’s something that it’s easy for them to imagine, and it’s something that it’s easy to give that one to five experiential rating about. So we’ve listed several other questions on the screen here, and that will be in the slides that, I believe, we’ll be sharing. I won’t go over all of them for the purposes of time, but we’ve modified every single one of the questions in the SUS to be more accessible and to take people with disabilities into account. So that their feedback can be better represented in this rating system.

ALWAR PILLAI: Today at Site Tech Global, we are really excited to say that we have made this freely available to everyone. You can go to www.makeitfable.com/aus, and you’ll be able to access s survey. You can fill out the form, and you will get an AUS and a report.

The reason we decided to do this is because we need to start improving research. We need to iterate on research, and this is Fable’s attempt at it. And we’re really excited, not only for you to use the survey, but also, to collect your feedback. Just like design and development is iterative, we think research should also be.

So please, go check it out. Try it out. The next time you are engaging with a participant who’s using assistive technology, you can share this with them, or you as a researcher can fill out the form on their behalf. The whole experience is accessible, and we’d love to hear your thoughts related to the questions or even related to the whole process.

So don’t hesitate to reach out to us and let us know your thoughts. There’s definitely a lot of work that needs to be done here when it comes to research and making it inclusive. We think this is a first step in the right direction, and with that, we have come to the end of our presentation.

We didn’t allocate time for questions, but we definitely do want to hear from you. And we want to have a conversation and an open conversation about how to make this more simpler, easier, and part of our entire process. So don’t hesitate to reach out to us at info@makeitfable.com, and we would love to chat with you about it. Sam, I’ll hand it over to you to do the closing before we hand it over to Robert.

SAMUEL PROULX: Absolutely. I am so incredibly excited to be able to begin having this conversation with everyone and that Fable has taken a step in the very important direction of including the voices of people with disabilities in user research. You know, this is a journey.

As Alwar said, there’s lots of iteration to be done, and there’s lots of steps to be taken. But this is an important one, so I’m really looking forward to being part of this conversation and to continuing with everyone forward on this ongoing journey. With that said, I hope that this presentation has been of interest to some of you and contains some tips that will help you on your research journeys as you go forward, making more accessible and more inclusive products. With that said, I will turn it back over to Robert.

ROBERT FRAWLEY: Great. Thank you, Sam and Alwar. This was a great session. For everyone currently on, be on the lookout for an email from Fable with the slides, as well as the survey link information. I will also be sending your Q&A’s to the Fable team as well, so we’re going to close out the session right now. But please make your way back to the main stage by going to sitetechglobal.com/event. Enjoy the show.