Usability Testing for Accessible Web Sites

by Anne Savitsky-Blondin

May 2006

Presentation 1: JAWS Navigation Study

To determine the correlation between design and navigation styles, and, ultimately performance, the team recruited 22 sight-impaired users who used JAWS independently, had a minimum of JAWS proficiency to avoid performance issues, had a moderate degree of Web expertise, used JAWS 4.5 or higher, and used IE 5.5 or higher. The users were further categorized into profiles— such as scientist, librarian, philosopher, and gambler— to determine how their core competencies affected their performance in navigating the site and completing tasks.

[The user profiles were what they discovered, not how they divided people in the first place. The whole point of the study was that they discovered the four types of behavior.]

[Minimum of JAWS proficiency? Most of them use it regularly.]

The users were assigned two tasks— calling a Fidelity representative about a college fund, and buying stocks using the Fidelity site— to complete on two prototypes of the Fidelity.com site, one which was Section 508 compliant, and the other, compliant with access keys, headings, and overviews, for which Ann explained there was no standard. The users were assessed by several factors: for example, their speed, reliance on information cues (such as headings), use of find/search features, use of navigational tools (for example, link lists), and use of structural elements on pages, such as tables and frames, to access information.

The team explained that performance for usability is a combined measure of speed and accuracy, otherwise expressed as task time + task success. The users in the study demonstrated increased performance due to Web expertise, use of headings, limited time on the home page, reduced need to read all material, and minimal use of the links list.

Of the users in the profile, the scientist demonstrated the fastest completion time (5:31), highest accuracy (93.3%), and least time spent on the home page (01:32), which may be attributable to their technical expertise, analytical approach, deductive reasoning skills, tendency to ignore content unrelated to their goal, and limited time spent listening to the home page in favor of scanning the overall organization and content, as demonstrated by their focus on specific page content as a last step.

In contrast, the gamblers completed the tasks the slowest (8:02) and with the lowest success (50.4%), a result consistent with lacking organization, haphazard navigation approach, and reliance on luck to get through links.

User behavior between the two versions of the site confirmed an important lesson for the team: Just because a site is technically accessible/508 compliant does not mean it is usable. Excessive use of links make the home page difficult to use, and could lead to cognitive overload. Further, headings are not required by Section 508 and shouldn't be relied on as a navigational tool. In the end, it is ok that a task takes longer if success is higher. They pointed to the study statistics to prove their point: Version 1 had an average task time of 0:06:43, with an average task success time of 76%. However, version 2, with the additional access keys and use of headings, fared with a slower task time of 0:07:49, but with a higher rate of task success of 83%.

The team noted some significant correlations between access keys and performance, including:

Increased use of the Read All mode increased time spent on a task and the home page
Frequent use of the Links List resulted in a slower task completion
Increased use of Ctrl Find resulted in reduced task completion, attributable to semantics and correct key word use
Increased JAWS expertise resulted in an increased use of headings for navigation
Increased reliance on headings increased the tendency to skip by content

The team suggested that the users' varied and unpredictable navigation methods demonstrate that design cannot be tailored to one demographic. They further advocated the importance of using universal design, on the following findings:

Knowing how JAWS users can, in theory, navigate a site is different than understanding how, in practice, they actually do.
Even knowing the range of JAWS commands available to users reveals little about how they string them together into larger navigation strategy when completing tasks online.

Some Recommendations:

To enhance the user experience for all, the team suggested the following design tips to ensure a usable site:

Use active terminology (for example, "View Account Balances" rather than "Account Balances")
Avoid link overload
Provide brief contextual overview for task completion. Cognitive overload reduces performance.
Design a consistence interface and provide a key so users know what is in it
Include collaborative learning opportunities, like discussion forums, for sharing useful information
Create useful search facilities, that help users find what they need using varied search terms. Users may also be confused between site-wide search and domain or database search (For example, Search Jobs)
Tell users who need accessibility features what you've got by including a page or topic they can find easily (for example, by searching on accessibility)

Continue to Part 3

Anne Savitsky-Blondin is an information architect specializing in content development for intranets and training.