Crafting a Learnable Experience: Redesigning Microsoft's Accessibility Insights for Web for novice accessibility developers

Context: Microsoft's Accessibility Insights for Web is a tool that aids web developers in evaluating and enhancing webpage accessibility through comprehensive manual and automatic tests.

Team: Microsoft Product Manager, 3 UX Designers, 1 UX Researcher

Valuable takeaways: I gained insights into conducting user testing with visually impaired users and learned about web accessibility and screen reader considerations.

Click for Figma Prototype.

Role: UX Research/Design

Timeline: 13 weeks

OUR PROBLEM

Novice accessibility developers found the tool confusing and difficult to navigate upon their first few uses.

OUR SOLUTION

55%

faster failure reporting of tests was achieved by

Enhancing the overview page layout by improving the data visualization and information hierarchy.

Improving the UX Copy of the onboarding page and streamlining the sidebar navigation by providing better information retrieval

To understand our solution better,
Let's backtrack a bit...

OUR PROMPT

We were tasked with redesigning the tool's features to enhance the user experience for novice accessibility developers.

Our industry partner, acting as the product manager, provided us with high-level insights from internal Microsoft research. Drawing from user research, we defined our project scope, the significance of our target users, and prioritized features to design.

OUR TARGET USER

We concluded that designing for novice accessibility developers would allow us to account for users with more knowledge.

User Characteristic

Frontend developers with little to no knowledge of accessibility

User Motivation

To start assessing websites for accessibility without prior knowledge​

User Selection Significance

To reduce the perceived complexity of web accessibility for frontend developers
OUR RESEARCH METHODS
Literature Review

I analyzed online reviews and existing literature to gain insights into usability issues within the tool. Our objective was to pinpoint the task flows and features most prone to usability issues.

Tool Walkthrough

Leveraging my background in Frontend development, I conducted a walkthrough of the overall assessment tool, performed assessments using the main assessment test, and annotated our findings on Figma. Our goal was to generate hypotheses about the task flows critical for novice developers and anticipate the challenges they might face.

Recruitment Survey

I assisted in formulating survey questions primarily aimed at recruiting developers for interviews with diverse levels of accessibility knowledge. Additionally, we sought to gather insights into the accessibility evaluation tools developers are utilizing for web assessment.

Competitive Analysis

A competitive analysis was conducted of assessment tools across various platforms and pricing models. Our goal was to gain an understanding of the strengths and limitations of other tools available in the market.

Learnability Study with Mixed Methods Think Aloud Approach

We conducted a learnability study with three trials lasting 30 minutes each for every participant, who were novice accessibility developers where they completed the same three tasks in all the trials. These three tasks were related to task flows critical for novice developers. The tasks are

Onboarding the tool and exploring its features

Conducting an assessment to get started

Reporting the accessibility issues

We recorded the time to complete each task, number of errors, user thoughts, and comments. SUS responses were collected at the end. A post-study semi structured interview was also conducted to clarify other data. Our aim was to measure and plot the learnability of the tool across trials. By employing a mixed methodology, we plotted time and error graphs, as well as SUS scores, and rationalized them with qualitative feedback from users. This allowed us to pinpoint the specific usability issues impacting learnability.

OUR REFINED PROBLEM STATEMENT

How can we improve the learnability of Microsoft’s Accessibility Insights for novice accessibility developers?

Learnability considers how easy it is for users to accomplish a task the first time they encounter the interface and how many repetitions it takes for them to become efficient at that task.

OUR FINDINGS

We consolidated our qualitative and quantitative findings and discovered that although the tool was reasonably learnable after two tries, there were usability issues affecting the efficiency of novice accessibility developers during their first use.

OUR IDEATION AND ITERATIONS

We conducted feedback sessions with developers with varying knowledge of accessibility and usability experts.

RESEARCH INSIGHTS AND OUR SOLUTION

Insight 1

Novice Accessibility Developers expressed that they wanted more guidance on which assessment to chose for their use case.

Insight 2

We found that users prioritized addressing the quickest or the most critical accessibility issues first, rather than attempting to find and fix all issues at once. However,

They struggled to promptly navigate to their required test category from the overview page and sidebar navigation.

Insight 3

Developers found that status of the tests was poorly reflected on the overview page and sidebar navigation.

Insight 4

Developers were not able to report the assessment progress sufficiently from the overview page.