You can get the Just Ask book from www.uiAccess.com/accessucd/print.html
Accessibility in User-Centered Design: Usability Testing
The Evaluating for Accessibility page provides guidance on incorporating accessibility into common evaluation methods, including standards review, heuristic evaluation, design walkthroughs, and informal evaluation with users with disabilities. This section is an overview of usability testing with participants with disabilities.
Usability testing provides quantitative and qualitative data from real users performing real tasks with a product. Usability professionals can evaluate some aspects of accessibility by using standard usability testing protocols, with a few modifications for including participants with disabilities.
You don't have to be a usability professional and you don't have to follow formal usability testing protocols to include people with disabilities in evaluation. Short informal evaluation can gather valuable feedback from people with disabilities without the rigor of formal usability testing. In most cases, including users in evaluation involves:
- finding a few people with disabilities,
- asking them to complete tasks on prototypes,
- observing them interact with the prototype,
- discussing accessibility issues with them.
While usability testing evaluates how usable accessibility solutions are by some people with disabilities, usability testing can't address all accessibility issues and doesn't evaluate conformance to accessibility standards. See Evaluating for Accessibility for guidance on incorporating accessibility into other evaluation methods, including standards reviews. It is most efficient to conduct other types of evaluations before investing in formal usability testing with people with disabilities.
In one case, an organization conducted usability testing of their educational software application with screen reader users. It was unusable with a major screen reader. Every user gave up in frustration and confusion within 20 minutes, after trying every strategy they knew to get at the content. The organization wasted £6000 on formal usability testing to figure out what they could have learned with easier, less expensive evaluation methods.
Conducting informal evaluations throughout a product development project is more effective than formal usability testing at the end of a project. While these chapters cover formal usability studies, much of the information applies to informal evaluations, field studies, and other collaboration with people with disabilities.
Some formal usability testing with participants with disabilities will use standard protocols. However, for usability tests designed specifically to explore accessibility issues, the protocol may be different from a typical general usability test; for example:
- you would likely use a think-out-loud technique with high facilitator interaction;
- data collection would focus on understanding errors related to accessibility issues, rather than on time-on-task or user satisfaction; and
- tasks would concentrate on specific areas of concern for potential accessibility barriers, rather than general site usage.
Detailed guidance on usability testing with participants with disabilities in the following sections:
- Planning Usability Testing covers determining participant characteristics, recruiting participants, choosing the best location, and scheduling the right amount of time.
- Preparing for Usability Testing covers ensuring the facility is accessible, preparing test materials, setting up and testing participants' configurations, becoming familiar with the assistive technology, and conducting pilot testing.
- Conducting Usability Testing covers setting up the room, orienting the participant, completing paperwork, completing the tasks, collecting data, providing compensation, and specific considerations for people with different disabilities.
- Reporting Usability Testing covers distinguishing between accessibility and usability issues, including relevant study parameters, being careful about categorizations and comparisons, clarifying conclusions, and writing about people with disabilities.
- Checklist for Usability Testing summarizes the tasks and considerations involved in planning, preparing for, conducting, and reporting usability studies with participants with disabilities.
- Recruiting Screener lists question to ask during recruiting.
The Interacting with People with Disabilities chapter provides additional information related to usability testing. The "Usability Evaluation, Usability Testing" section of the Appendix: Resources includes references that provide general information on usability evaluation and usability testing.