A/B Testing
Why?
A minor change in a design may alter user behaviour in ways that are hard to detect in a usability test. An A/B test allows you to compare real-world user behaviour across different versions of the product.
How?
In an A/B test the user is confronted with one or two versions of an interactive product. Remote software records metrics for user behaviour. These metrics are compared between the two versions to see which alternative is better. Sometimes more than two alternatives are tested: A/B/X testing.
Ingrediënts
- Specialized software.
- Reliable metrics.
- A large amount of users.
In practice
Google does a lot of A/B testing on its search services. The tiniest details can count for Google because they have so many users. Some companies also use the A/B experimental design for testing concepts, rather than finished products.
- Universal Methods of Design (Hanington & Martin, 2012). Spends a couple of pages about A/B tests. In the Dutch version chapter 1. P8-9
- A good background read for doing an A/B test can be chapter 9 of Researching Information Systems and Computing (Oates, 2006) on experimental design p126-140
- This video by the Nielsen Norman Group explains how to plan a systematic set of A/B tests