Accessibility in User-Centered Design: Planning Usability Testing

The Evaluating for Accessibility page provides guidance on incorporating accessibility into common evaluation methods, including standards review, heuristic evaluation, design walkthroughs, and informal evaluation with users with disabilities. The Usability Testing section is an overview of usability testing with participants with disabilities.

Planning a usability study that includes people with disabilities involves the following considerations:

• Determining Participant Characteristics
• Recruiting Participants with Disabilities
• Choosing the Best Location
• Scheduling the Right Amount of Time

Determining Participant Characteristics

There is no definitive answer to the questions, "How many participants with disabilities should be included in usability testing?" and "What characteristics should they have?" It depends. This section explains factors that can help answer those questions for your particular situation.

Understanding the challenge

The number of participants to include in any usability test is a subject of debate among usability professionals. [1] Some studies have led to the conclusion that testing a large number of participants does not yield significantly more information than testing fewer participants, since the first participants will find most of the usability problems. [2] [3] Some research shows that five participants is sufficient to find 85% of the usability issues when you have comparable users who will be using the product in fairly similar ways. When you have several highly distinct groups of users, you need to test additional users. [4]

People with disabilities use products differently, and so a thorough usability test for accessibility requires more than five users. (The next section includes guidance for using fewer than five participants with disabilities.) When you have different categories of users, one recommendation is to include three users from each category. [4] However, people with disabilities do not fit easily into categories in terms of product interaction.

Careful with categorization

Disabilities are sometimes grouped into four high-level categories: visual, auditory, physical, cognitive; yet there is vast variability within each category. For example, people with visual disabilities include a middle-aged woman who has low vision since birth and is very experienced with screen magnification software, a young man who recently went totally blind from retinitis pigmentosa and is a new screen reader user, an elderly woman whose sight is deteriorating from macular degeneration yet she doesn't use any assistive technology, and a young boy whose color blindness has not yet been diagnosed.

This variability within categories is significant in product design and evaluation. For example, although visual disabilities are often categorized together, a product can be accessible and usable for a person who is blind, and yet totally inaccessible and unusable for a person with low vision, and vice versa.

We evaluated a website that worked great for screen reader users, but was terrible for people with low vision who use other adaptive strategies, and for people with somes types of cognitive disabilities.

How a person interacts with products is also impacted by differences such as: whether the person was born with the disability, acquired the disability at a young age, or acquired it when older; whether the disability is temporary, permanent, static, progressive, or regressive; and what adaptive strategies and assistive technology the person uses. Also note that it is common for people to have multiple disabilities across more than one category.

Because of this variability within a category, the four common categories of disability (visual, auditory, physical, cognitive) are not sufficient for categorizing usability study participant characteristics.

We wanted a definition of blindness for recruiting for a formal study on people using the Web. We talked to a specialist at a leading vision institute who said that the World Health Organization had not yet come to an agreement on a definition of blindness. He suggested that we recruit by what assistive technology potential participants use, rather than a definition of blindness. This turned out to be excellent advice for this study and a subsequent study of people with low vision. We had one participant who was legally blind by our country's definition of blindness, and yet he had enough vision that he usually used screen magnification software rather than a screen reader. Thus, he fit better in a study of screen magnification users than he would have in a study of users who are blind.

Identifying a realistic range of participants

Projects rarely have the time and money resources to do thorough usability testing with a wide range of participants with disabilities. The number of usability test participants with disabilities included in a given usability test is usually determined by limited project resources. The following considerations can help you make the most of limited resources and focus on which participant characteristics to include in your usability tests.

Understand the overlap between accessibility and usability. In deciding the number of participants with disabilities to include, consider that users with disabilities also address general usability issues that impact all users, including users without disabilities. Therefore, usability testing with participants with disabilities will identify both accessibility issues and general usability issues. In many cases general usability issues are amplified when testing with participants with disabilities, making it easier to find issues that impact all users.

Explaining how users with disabilities also address general usability can help you get more time and money budgeted to include users with disabilities in usability testing and throughout your project.

Start with standards reviews, heuristic evaluations, design walkthroughs, and screening techniques. Conducting other evaluation methods first will identify many accessibility issues that you can fix before bringing in people with disabilities. Also, the other evaluation methods can help you focus on that is most important to test with real people with disabilities.

Screening techniques can effectively supplement testing with people with disabilities in some cases. For example, anyone can test using a cell phone with one hand; you don’t have to have a disability to do that. However, some aspects can’t be effectively evaluated with screening techniques. For example, if you don’t have much experience with screen readers, you won’t be able to effectively test your product with one.

Include different characteristics in different usability tests. Ideally, product designs go through several iterations of usability testing. In some cases you can include participants with different characteristics in each round of testing, rather than trying to fit participants representing all characteristics into every test.

Focus on the target users. If your target users include a higher percentage of people with a certain disability, focus on those relevant characteristics. For example, a usability test for a product that is marketed primarily to seniors should include seniors who have age-related disabilities; and a test for a website containing information on diabetes should include people with visual disabilities. Note the important cautions below.

Focus on highest impact. Some products will impact people with a specific disability more than people with another disability. You might not need to include participants with certain characteristics because of the nature of the product or the situation. For example, if you're designing a product that produces no sound, you might choose not to actively recruit participants who are hard of hearing; if you are designing an intranet (internal website) and the company standard is a specific screen reader, you might choose not to include participants who use other screen readers. Note the important cautions below.

Important cautions

Avoid the pitfall that one participant represents all, as explained in "Individual Differences" in the Analysis Phase chapter.

Avoid the assumption that people with disabilities can't or don't use a product. For example, don't assume that a person who is blind would not want to use a drive-up ATM, research cars on the Web, or use a kiosk to renew license plates. Even though a person who is blind does not drive, he might want to use a drive-up ATM from a taxi cab on the way to the airport, research cars on the Web as a gift for a child graduating from university, or use a kiosk to renew license plates that are registered jointly with a spouse who drives. See the "Accessibility Considerations for User Group Profiles" section of User Group Profiles for more about this caution.

A friend of mine who is blind wanted to research driving license requirements on the Australian Traffic Authority website so he could discuss the process with his daughter.

Assistive technology (AT)

Assistive technology (AT) is a significant variable in determining participant characteristics, particularly for software and web products. Different assistive technologies can interact differently with a given product; for example, different screen readers interact differently with the same application. Therefore, it is usually best to include multiple screen readers and multiple versions of the same screen reader in your evaluations.

Some assistive technologies are complicated and difficult to learn, and so the participant's experience and skill with particular AT impacts the usability test. A user with insufficient skill may not know how to use the AT effectively, and spend lots of time figuring it out rather than using the product being tested. Also, problems identified might not be the fault of the product, rather they could be because the participant doesn't know how to use AT effectively. On the other hand, an expert AT user might know uncommon work-arounds to overcome problems that the average user would not be able to handle.

In one case, a very knowledgeable screen reader user examined the HTML code of the web page to find a way around an accessibility barrier.

Just as with any evaluation with users, whether you include novice, average, or advanced users depends on your target users. For example, if you're developing an application to be used by accountants inside a company, you probably want advanced AT users; for a public website to apply for disability benefits, you want novice AT users. For most usability tests, use participants who have medium to high skill with their AT. Questions to find out about a potential participant's assistive technology experience and skill are included in the Recruiting Screener near the end of the book.

Occasionally a user we had profiled as experienced turns out to have used their AT for a long time, but use only a fraction of its capabilities. This could have been a useful test case in itself, but not when we already had enough beginner users lined up. As a result, we now recruit for expertise rather than experience. Even then, we find that users often overestimate their skill level.

Recruiting Participants with Disabilities

Often usability specialists think it will be difficult to recruit people with disabilities, yet after a little bit of effort, it turns out to be fairly easy.

Plan for additional recruiting. With any usability test, the more specific the participant requirements, the longer it takes to recruit. For example, recruiting doctors age 35-55 who use Macs takes longer than if the only requirement is that participants are age 35-55. The same is true of recruiting people with disabilities. Adding specific participant requirements, such as type of disability and use of assistive technology, will likely add time and effort for recruiting.

On the other hand, your recruiting could take less time than usual because of "viral marketing" within disability communities.

People with disabilities have responded to recruitment flyers more quickly, in greater numbers, and with more persistence than others. We've found that generally people with disabilities are more willing to participate in studies, particularly if it is in support of making a product more accessible. Also, our standard honorarium seems to be more of a draw, as many people with disabilities are underemployed.

Make key contacts. Places to look for participants with disabilities include the following:

• Social media groups
• Organizations for specific disabilities or conditions
• Cross-disability organizations
• Mailing lists
• College and university programs for students with disabilities
• Local disability-related support groups
• Local or regional government rehabilitation or disability services departments
• Seniors organizations and local senior centers
• Independent living organizations

Searching the Web for "disability organizations" returns links to several lists. Include your country, state, or locale to get more localized results.

Some groups have newsletters and mailing lists where you can send a recruiting announcement.

A disability organization sent an e-mail to all its members asking those who met our recruiting criteria to contact me.

Many national and international organizations have regional contacts who can help find local participants.

When we briefly explain what we're doing to a contact at a disability organization, we usually get very positive response and help recruiting. Also, when we were first starting to include people with disabilities in our studies, we got some very useful tips from asking, “Do you have any suggestions for us to help people feel comfortable participating in the study?”

Finding a person within an organization who is interested in helping can save a lot of time.

I contacted the services coordinator at a local senior center and told her about our project. She was eager to help and recommended potential participants based on the characteristics that we were looking for. Her help saved a lot of screening time and effort.

Consider pilot tests as a recruiting tool. Some people with disabilities are part of networks that actively share accessibility information. Word of a positive usability testing experience can spread rapidly and result in potential participants contacting you.

For one study we recruited participants from a technical college program for students with disabilities. The first day we conducted 2 tests with students with disabilities. When we returned to the college the next day for 2 more scheduled tests, the news had spread and 12 more students with disabilities were lined up wanting to participate.

To take advantage of this word-of-mouth recruiting, consider conducting pilot tests early in your recruiting efforts and seek out participants who are well-respected, vocal members of a community, such as a student organization, senior center, or disability support group. Encourage them to spread the word of your recruiting.

Add relevant information to recruiting screener. When screening participants with disabilities for a usability test, use the same parameters as you would for participants without disabilities. Additional information to cover in the recruiting screener includes technology use, alternative formats for printed materials, etc. Specific questions to add to the screener are included in the Recruiting Screener section.

Arrange for interpreters as needed. If you're conducting usability studies with participants who are deaf, it's customary for you to find, schedule, and pay for sign language interpreters. (This is usually not the case when a person with a physical disability has a personal assistant; they will make the arrangements.)

Plan to reimburse participants for necessary expenses. Although the basic compensation for people with disabilities is the same as for any participant in your test, there might be some additional cost considerations. Transportation might be more complicated and costly for participants with disabilities. You should usually reimburse for additional transportation expenses, such as an accessible taxi van.

For some participants you might need to pay for the time of a personal assistant or interpreter. As part of the recruitment process, find out about participant expenses and confirm with participants which you will reimburse.

We arranged with a corporate taxi service to drive participants between the closest public transportation stop and our offices. We worked with the taxi company to ensure a safe and comfortable experience for participants.

Choosing the Best Location

With any usability test there are advantages and disadvantages of conducting the sessions in a usability lab or "in the field" at the participant's work, home, or other location. Some people use remote evaluation where the facilitators and participants are in different locations. When conducting usability tests with people with disabilities, there are additional factors to consider when making the decision of location.

Consider the goals of the usability test. Where is the best location depends partly on the goals and potential additional benefits of the test.

Most designers don't know how people with disabilities use their products, and many people are uncomfortable around people with disabilities. Usability testing can be a great way to change that, as described in the Involving People with Disabilities in Your Project chapter.

Conducting usability tests in your lab usually provides the opportunity for more of the project team to observe, and for you to record the sessions more easily. Informal sessions with lots of interaction between designers and participants are especially useful, and can be conducted wherever is most convenient for the project team, rather than in a formal lab.

Our project manager let us conduct usability tests with people with disabilities; however, she wasn't convinced of the value. We wanted her to be able to observe some of the sessions, so we conducted them where it was convenient for her.

When the goal is for a couple of people to learn more about how people use the product in their own environment, field studies are best.

When you want face-to-face interaction, remote evaluation is not an option. Additionally, remote evaluation doesn't provide equivalent results as in-person usability testing. However, there are cases where remote evaluation is best. For example, if you already have an established relationship with a person who helps you evaluate for accessibility, it may be equally effective and much easier to do some evaluation remotely rather than traveling to the same location.

We set up remote evaluation with an expert screen reader user who we work with often. In order for us to see his screen, he needed to designate himself as "meeting manager"; however, that part of the software was not accessible—his screen reader didn't see it at all. The software did have keyboard shortcuts so we were able to tell him how to set it up. Once that was set, the rest of the interface was accessible through his screen reader. Then anyone in our organization could log on and see his screen, and hear his voice and screen reader through a standard speaker phone on his end and teleconference bridge on our end.

Consider assistive technology needs. Assistive technologies are a significant factor in where to conduct usability tests.

Providing the required assistive technologies in a lab can be costly, complicated, and time-consuming. When testing software or web-based products, participants might require different versions of assistive technologies with different configurations. Changing system configurations between participants can be difficult.

It is common for an information technology specialist, rehabilitation specialist, or family member to set up a person's home or office computer system, including assistive technologies. Some assistive technologies have many settings; for example, screen magnification software provides multiple options for zoom, color, contrast, cursor, and more. Participants might not know how to configure assistive technologies in your lab to work as they are used to them working in their regular system.

Many of the participants in a study with screen magnification users could not configure the software themselves; however, they could describe how it usually looked and worked for them. We had someone who was familiar with the software settings work with the participants at the beginning of the session to get it set up as they wanted.

Some participants might be able to bring their assistive technology with them to your lab; for example, bring their own laptop with screen magnification software. This could work well for a study of a site on the Web with manual data recording; however, it probably would not work well for a study of your beta software product using data recording software.

Carefully consider potential complications before loading software onto a participant's computer, or asking them to download and install software. Some assistive technologies don't play nicely with other software and you don't want to be responsible for causing serious problems with a participant's system. This is an issue when people bring their own laptop to your lab, as well as for remote evaluation.

Take into account transportation. Transportation may be more difficult for some participants with disabilities. Also, accessible transportation is often unreliable and participants using such transportation might not be able to meet a tight schedule in a usability lab.

We conducted our sessions on a university campus, rather than our lab. The campus was much better for participants using public transportation, and that made recruiting easier.

Evaluate the accessibility of potential locations. Many buildings that claim to be accessible could have barriers that make it difficult for people with disabilities to participate in the test. For example, one site that was technically wheelchair accessible had very thick carpeting that made access with a manual wheelchair difficult. "Ensuring the Facility is Accessible" in the Preparing for Usability Testing section provides specific points to consider regarding the location's accessibility.

Scheduling the Right Amount of Time

Use pilot tests to work out timing. Plan extra time for your pilot tests and record how long each step takes.

In one of our first studies with people with disabilities, we planned to have participants complete the consent form, web expertise survey, financial knowledge assessment, and post-study survey online. However, our first pilot participant with low vision took almost 1 hour of the planned 1.5 hour session to complete the online forms. Thus, we dropped the web and financial surveys from the study.

Plan based on specific disability considerations. In some cases, the time for each usability test participant session will be impacted by a participant's disability and longer or shorter sessions may be more effective. For example, people with some physical disabilities, such as tremor and poor fine motor control, might take longer to do basic task steps such as activate a button, and a person with a cognitive disability might need longer to process textual information and instructions. It will take some people with disabilities longer to complete tasks, especially if the product is not highly accessible or if they haven't used products like it before.

You may need to schedule additional time between sessions as well. It may take some participants longer to complete pre-test paperwork, get in and out of the testing room, and take breaks. You might also want to plan for an additional staff person to be available, for example, to help escort participants around the facility.

Participants have asked our staff to walk their guide dogs during sessions.

Be aware of energy level considerations. Fatigue tends to be more of an issue with some participants with disabilities due to factors such as the disability itself, medications, and the extra effort required to use assistive technology. Participants might need breaks and might not be comfortable or effective in long sessions. Ask participants during recruiting about any time and fatigue considerations, and plan session timing accordingly. A sample question is included in the Recruiting Screener.

A participant using a screen magnifier told us that he gets nauseous after a half-hour of watching the screen and would need a short break every 30 minutes.

On the other hand, some participants with disabilities have high energy levels and will be effective longer than some people without disabilities. Some are used to taking longer to accomplish tasks and have more patience and more determination to complete a task successfully. They might want extra time.

Plan for different situations. For example, have a shorter task list to start with and have additional tasks ready in case you and the participant want to do more.

Schedule time to confirm assistive technology setup. Plan time at the beginning of each test session for participants to check that any assistive technologies are set up and configured as they want.

Plan time for the participant to become familiar with your product. When your usability goals are for people somewhat familiar with your product, as discussed in "Setting Usability Goals" in the Analysis Phase chapter, the usability testing schedule might include time for the participant to interact with your product before usability testing begins.

Now let's talk about preparing for usability testing with participants with disabilities.

Navigation

• Welcome
• Cover
• About
• Table of Contents
• Introduction
• The Basics
  • Incorporating
  • Involving
  • Interacting
• Accessibility in UCD
  • Background
  • Analysis
  • Design
  • Evaluation
    • Screening Techniques
    • Usability Testing
      • Planning
      • Preparing
      • Conducting
      • Reporting
      • Checklist
      • Recruiting Screener
• Resources
• Updates

Document Information

Excerpt from Just Ask: Integrating Accessibility Throughout Design by Shawn Lawton Henry. www.uiAccess.com/accessucd/

Copyright © 2003-2020 Shawn Lawton Henry, All Rights Reserved.

Contact. www.uiAccess.com/accessucd/contact.html

Sponsors

Thank You Just Ask Sponsors:

• 
• 
• 
•