5.7.5 Keeping Variability Down

As we've seen, your ability to make good estimates based on bottom-line test results depends on the results not being too variable. There are things you can do to help, though these may also make your test less realistic and hence a less good guide to what will happen with your design in real life. Differences among test users is one source of variable results: if test users differ a lot in how much they know about the task or about the system you can expect their time and error scores to be quite different. You can try to recruit test users with more similar background, and you can try to brief test users to bring them close to some common level of preparation for their tasks.

Differences in procedure, how you actually conduct the test, will also add to variability. If you help some test users more than others, for example, you are asking for trouble. This reinforces the need to make careful plans about what kind of assistance you will provide. Finally, if people don't understand what they are doing your variability will increase. Make your instructions to test users and your task descriptions as clear as you can.