Communications of the ACM
Number of People Required For Usability Evaluation: The 10±2 Rule
Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users.
The full text of this article is premium content
Comments
CACM Administrator
The following letter was published in the Letters to the Editor in the August 2010 CACM (http://cacm.acm.org/magazines/2010/8/96618).
--CACM Administrator
No usability conference is complete without at least one heated debate on participant-group size for usability testing. Though Wonil Hwang's and Gavriel Salvendy's article "Number of People Required for Usability Evaluation: The 102 Rule" (Virtual Extension, May 2010) was timely, it did not address several important issues concerning numbers of study participants:
Most important, the size of a participant group depends on the purpose of the test. For example, two or three participants should be included if the main goal is political, aiming to, say, demonstrate to skeptical stakeholders that their product has serious usability problems and usability testing can find some of them. Four to eight participants should be included if the aim is to drive a useful iterative cycle: Find serious problems, correct them, find more serious problems.
Never expect a usability test to find all problems. CUE-studies(1) show it is impossible or infeasible to find all problems in a Web site or product; the number is huge, likely in the thousands. This limitation has important implications on the size of a test group. So go for a small number of participants, using them to drive a useful iterative cycle where the low-hanging fruit is picked/fixed in each cycle.
Finally, the number and quality of usability test moderators affects results more than the number of test participants.
In addition, from a recent discussion with the authors, I now understand that the published research in the article was carried out in 2004 or earlier and the article was submitted for publication in 2006 and accepted in 2008. All references in the article are from 2004 or earlier. The authors directed my questions to the first author's Ph.D. dissertation, which was not, however, included in the article's references and is apparently not available.
Rolf Molich
Stenlse, Denmark
REFERENCES
(1) Molich, R. and Dumas, J. Comparative usability evaluation (CUE-4). Behaviour & Information Technology 27, 3 (May 2008), 263281.
Displaying 1 comment