How many players should you playtest with?

So, you’ve started developing your game, and you’ve got your basic gameplay done. It’s time to start playtesting*. The question occurs to you: how many people should be playtesting my game?

*If you’ve got something you can playtest with, you should be playtesting. As soon as possible. As I’ll explain in a bit, it’s really important to plan your playtesting (and we’ll be exploring this more in the coming weeks), but you really want to get to it pronto.

Jakob Nielsen, usability king, says that you only need 5 users to test each iteration of a design, and in many ways, this philosophy applies directly to playtesting. Yet this only truly applies to early phases of playtesting: in the latter stages, we’ll be taking a different approach.

Nielsen’s basic argument is that testing is about improving a design rather than finding every single problem. Once you get past 3-5 users, each new user you add to your test tells you less and less about how to improve your design, simply because problems get hidden behind other problems. The answer is to save your time and fresh playtesters (both of which are limited resources) to run more iterations of playtesting: in each iteration, 3-5 is enough.

There’s a caviet to this: if you’re just sending your game to 5 friends and asking them to ‘tell you what they think’, 3-5 probably isn’t enough. In fact, if you’re not planning your playtesting at all, the number probably doesn’t matter: you still won’t get the data you need. The full process of designing and running a test is a far bigger topic than this post, and I’ll be exploring it fully over the next few months. For now, your best course of action is to:

• Sit next to the person playtesting your game
• Ask them what they think of each level or game element as it appears
• Observe (do NOT talk to the player - let them play and don’t interfere) and record the issues you find, as well as any important factors (e.g. play length, number of deaths, frustration and emotional response)
• Do this with a few people, then fix the issues you think you need to.
• Repeat

There are ways of getting almost as much information from your testers in an online playtest. Where possible, however, being in the room is better.

There are two exceptions to this rule of 3-5. The first is if you’re doing a comparative study (comparing two slightly different systems to see which one is ‘better’), testing multiplayer, or have two types of players you’d like to test with (such as expert and novice). We’ve just put together our Stage 1 testing plan for Particulars, and we’ll be starting with an 8 player playtest: we’ll be getting 2 players at a time to do a single player playtest simultaneously, before doing multiplayer with the two of them. The single player test will serve as a way of training each player, as we’re more interested in the multiplayer interaction than in learning in a multiplayer environment at this stage of testing.

The other exception is in mid-late playtesting (also known as beta testing), when a larger number is a necessity. Why? Because the aim of playtesting isn’t completely design-oriented. Games are rather hard to test for correctness, and so it’s good to have a large number of people playing your game before it comes out, probing all the cracks for those nasty little bugs that can ruin your day. At this stage, you’re also looking for those really subtle design issues that won’t come up in a small test, but a large number might pick up.

So that’s it. Playtest early, playtest thoroughly and playtest often. Until you’re much further on, however, you don’t need to playtest with lots of people.