According to recent reports, kate Kalcevich has over 20 years of experience turning accessibility into a driver of innovation and growth. As VP of Innovation at Fable, she oversees a … More about Kate ↬.

Weekly tips on front-end & UX.Trusted by 182,000+ folks.

In the summer of 2024, I became co-chair of a working group of expert researchers who came together to determine how best to perform accessibility testing with people with cognitive disabilities. This was work I did for Fable, where I am right now VP of Innovation.

According to recent reports, cognitive disability is an umbrella term for several disabilities that impact how people process information, and it usually affects memory, focus, and/or learning. It is the most prevalent disability in the U.S. (13.9% via CDC), and cognitive disability is increasing rapidly (Yale study).

We set four goals for ourselves to learn how to work with this audience:.

We created a screener to recruit people who self-identified as having challenges with memory, focus, and learning. We also reviewed published studies that involved cognitive testers to learn best practices for working with them.

According to recent reports, next, we tested these best practices with an initial group of 25 testers in a pilot study. We fine-tuned our approach iteratively and created a guide to running user interviews with cognitive testers and a survey that could quantify their experiences using digital products. Finally, we documented what we learned.

After our pilot study with this fresh group of testers finished, I felt that they would uncover more usability insights than the general population (gen pop) user research participants I’d worked with in the past. I set out to validate this hunch.

I decided to run a joint study with Fable’s partners at the University of California, Irvine, in collaboration with Syed Fatiul Huq and with help from Fable researchers Pranav Pidathala, Ali Brown, and Michael Fagan to see if my hypothesis about finding more insights with cognitive testers proved true or not.

Industry observers note that I generated three websites for the study using an AI prototyping tool. I wanted three different types of sites with different user goals and content so I could test a variety of tasks in the study.

We used a single screener with questions about memory, focus, and learning, and screened participants into two groups based on whether they self-identified as having cognitive challenges or not.

We ran 30 user interviews, 10 per website, with an even 5⁄5 split between cognitive and gen pop participants for each website. In each session, a participant completed all the tasks for one website during an online user interview facilitated by one of the researchers involved in the study.

Recent findings suggest that all participants completed an Accessible Usability Scale (AUS) survey at the end of their session. This is a free, Creative Commons-licensed 10-question survey to evaluate the usability of websites and mobile apps.

Industry observers note that I reviewed all the study recordings and transcripts and made note of every time a participant raised a concern, question, difficulty, or asked a question about how something worked. I counted all of these as complications. I also noted where a participant missed something that was part of a task, even if they didn’t notice it themselves. I also noted every suggestion for improvement made by participants.

Examples of issues found included:.

Examples of suggestions included:.

Problems and suggestions were counted once per participant, even if they mentioned the same thing twice, but there are, of course, repeat issues and suggestions across the different participants. It is expected in UX research with multiple participants that you’ll find similar issues with each participant, and that is a signal that an issue is a universal challenge.

Across the three websites tested:.

The results aligned with my instincts: participants with cognitive disabilities identified 1.8 times more issues and made 1.8 times more suggestions than gen pop participants.

Let’s dive deeper into the data for each website. Note that an AUS score ranges from 0 to 100, with higher numbers representing better usability than lower numbers.

This site had the simplest design and content of all websites tested in the study and accordingly had the lowest overall issues and the highest median AUS scores. The data aligns with what you’d expect from an easy-to-use and simple website.

According to recent reports, on this website, cognitive participants found 3.4 more problems and made 2.2 more suggestions on average. Their average score of the overall experience was 13.7 points lower than that of the gen pop participants.

This was the website with the most varied functionality and the most tasks to complete (4), so it’s not surprising that participants found the most complications.

Here, cognitive participants found 6 more concerns and made 3.2 more suggestions on average. They also scored the overall experience 17.2 points lower than gen pop participants on average.

This website was intentionally designed to be complex, and task 3, finding the bridal package, was meant to be extremely difficult to complete.

Sources indicate that on this last website, cognitive participants on average found 7 more issues and made 2.4 more suggestions. Their average score for the overall experience was 14.3 points higher than the gen pop participants.

Sources indicate that something interesting happened with the AUS scores for cognitive and gen pop participants in Tables 3 and 4. Cognitive participants scored Crown & Comb higher than Turning Pages, but gen pop scored the opposite — higher for Turning Pages and lower for Crown & Comb. If I had to guess why, I suspect finding more issues on Turning Pages impacted the cognitive participants’ perceptions of usability more than the gen pop participants’.

The other major difference between the sites, outlined in Table 5 below, was that cognitive participants found many more issues with buttons and links on Turning Pages and more issues with icons and visual elements on Crown & Comb. This suggests to me that the interactions being challenging on Turning Pages were a more significant challenge than issues with visual elements.

When it comes to the more qualitative findings, I looked at trends in the types of concerns found by both groups of participants.

Sources indicate that when I grouped issues by category, the following issues surfaced more often with cognitive participants: content, buttons and links (affordances and function), icons or visual elements, and media (video, animations). They nearly tied with gen pop participants on navigation issues (45 vs 46).

Let’s look at the commentary provided by one cognitive participant versus one gen pop participant in the Crown & Comb sessions. The cognitive participant gave an AUS score of 38, and the gen pop participant gave an AUS score of 27.5. I chose to compare these two participants because they both gave the lowest scores within their group.

Sources indicate that notice the differences in how they described the overall experience in the quotes below. The gen pop participant explained it was frustrating and not engaging. The cognitive participant felt drained and less able to focus. I interpreted the experience as having a more profound impact on the cognitive participant’s overall wellbeing.

In summary, across all 3 websites we tested, participants with cognitive accessibility needs identified 197 usability concerns, compared with 113 identified by gen pop participants.

Cognitive participants made 93 suggestions for improving the user experience, compared with 54 suggestions by gen pop participants.

When I compared complications and suggestions across both groups of participants, it turned out that the cognitive participants found 1.8 times more issues and made 1.8 times more suggestions than gen pop participants.

Cognitive participants surfaced more challenges related to content, buttons, icons, visual elements, and media than gen pop participants.

In working with cognitive participants for the last few years, I’ve seen how they surface cognitive load concerns consistently. These issues don’t just impact people with cognitive disabilities such as neurodivergence; they also impact:.

What would I have missed if I hadn’t included cognitive participants, and how might that have impacted the business outcomes for these websites?

On the Strong Snacks website, the cognitive participants surfaced:.

These are improvements that would give all people more trust in the content while also making it easier to read and skim for key content. The research findings point towards design best practices, such as not having continuous animation and using layout to draw attention to different types of content that a senior designer might also point out.

Without cognitive participants, we might have missed the more subtle but important issues with confusing interactions, such as how the “Add to book bag” button worked. They were also confused about where reviews and recommendations came from. Both of these issues could decrease a user’s trust in the website.

All participants surfaced that the book-matching feature was hard to find, but the deeper problem the cognitive participants emphasized is that the site’s interactions don’t consistently behave in ways that they can predict and understand, decreasing their confidence.

In a recent development, anyone who wants to buy a book could benefit from a clear understanding of how to add books to a cart and complete the checkout quickly and with no ambiguity. Compounded over hundreds or thousands of people, a lack of clarity in a purchase flow will lead to lost revenue.

The Crown & Comb website in particular highlighted the benefits of having cognitive participants who raised:.

These complications likely also affect gen pop participants, but they are more likely to muddle through a task with incomplete information. However, that can lead to losing customers to a better experience if a competitor pops up. Loyalty is often tied to experiences, not just brands, and having a poor experience means your customer retention can be weaker.

It has been revealed that the study revealed that finding a bridal package was hard for everyone, but the cognitive group showed how that became an accessibility barrier. When you combine:.

Sources indicate that you create a high enough cognitive load that some people will not be able to complete the task. In my opinion, this is where usability issues start to become accessibility barriers — when they increase cognitive load so much that it becomes overwhelming for some users.

The percentage of people aged 65 and older in America is projected to increase from 17% to 25%. By 2060, 1 in 4 Americans will be an older adult (U.S. Census). This is where everyone starts to experience cognitive decline. As the aging and cognitive population segment expands, companies will need to build for these more complex user needs.

Recent findings suggest that people with cognitive access needs are a natural starting point because they will find the types of usability concerns that UX teams are used to. This could make cognitive an easier entry point for inclusive research. Getting insights from assistive technology consumers is still very important, but many teams don’t know how to start doing that.

Cognitive accessibility is a powerful on-ramp into broader accessibility research and testing. By focusing first on cognitive load, clarity, and predictability, we build research foundations that make future work on accessibility with screen readers, screen magnifiers, and alternative navigation users more approachable.

In this small exploratory study, participants with cognitive disabilities identified 1.8 times more issues and made 1.8 times more suggestions than gen pop participants. I’ve seen this type of impact in research conducted by Fable customers’ websites that aren’t AI-generated, too.

Cognitive inclusion in UX research is not optional, and it’s not just about accessibility. It’s how UX teams can make their research more efficient, create clearer content, simpler flows, and ship better products for everyone.

Sources indicate that this study had a relatively small sample size, so the findings are more qualitative than quantitatively validated. Testing was also done on two different platforms. Cognitive participant sessions were run using Fable Engage, and gen pop sessions were run on UserFeel. Different platforms with unique participant panels can affect the quality of insights and comfort levels with user research participation.

Disclosure: I work for Fable and chose to use our platform because it was more affordable than paying for access to another research platform, allowing me to include more participants in the study at a lower cost.

It has been revealed that different researchers facilitated the user interviews, which can also affect findings, but all sessions used the same task structure and discussion guide template, and all were completed online. Even though the sessions were facilitated by different researchers, the issue and suggestion counts were all done by me to ensure consistency across all websites and participants.

I’ve compiled a few useful resources as you begin your cognitive inclusion journey.

In a recent development, tips on front-end & UX, delivered weekly in your inbox. Just the things you can actually use.

With practical takeaways, live sessions, video recordings and a friendly Q&A.

Industry observers note that everything TypeScript, with code walkthroughs and examples. And other printed books.

→ Read original article