This month’s article from HFI is about how many participants to involve in a usability test. And the answer is [insert drum roll here] 12 per user segment. Yay!
No matter what the answer is (Spool, Nielsen), the more important issue is that the question is wrong. The discussion about how many participants to include in a usability test is based on the premise that the ultimate goal of running a test is to identify as many usability issues as possible.
This is not the goal of a usability test. It’s not even a sensible or realistic goal. Even if we could identify every usability problem, by the time we fix all of them, we will have introduced new ones. Then we find all of the problems again, fix, introduce new ones, and so on until only a very small number of usability problems remain.
In reality, the most important goal of a usability test is to identify the main usability problems – the ones that affect all users, are high impact or high risk. We redesign those and test again, but with a smaller focus to start with, we can get to a good product more quickly.
As I mentioned a few days ago, big-bang testing is not the right way to usability test anyway. Usability testing is most useful as part of an iterative, user-centred design cycle. If you usability test as a part of the design process rather than as a scientific experiment, you will have happy customers instead of statistical significance…I know what I’d prefer.