Playtesting 103: What to Measure

August 02, 2011

Originally posted on Gamasutra

This is part 3 in a series on how to playtest games (click through to parts 1 2 4 5 6).

The next step in our playtesting journey is deciding what you should measure. Measurement comes in many forms, and so today we’ll mostly be looking at what you’ll be measuring. We’ll look at how to collect that data next time.

 Measuring during playtesting is pretty damned important. It’s one of the major differentiators between a good playtest and a bad one – you can set everything up perfectly, but if you aren’t collecting the right data, you’ll miss something important.

The data you need will (shock horror!) depend a lot on what your test purpose is. As a general rule, early stage, on-location testing will benefit more from qualitative data (partly because it’s harder to come to conclusions from small amounts of quantitative data with a small sample), while later stage, online testing will benefit more from quantitative data (because wading through that qualitative data will take much more time).

This isn’t to say that you should have only qualitative or only quantitative data at any point: there’s no point at which a ‘What was your least/favourite part’ question will be wasted, and no point at which monitoring how many deaths won’t be supremely useful. In the end, you’ll probably want a mix of each.

So, now for a long list of things you might want to measure, starting with the qualitative:

…and continuing with the quantitative:

What you’ll notice from these two lists is that many of the qualitative data can be covered, at least partially, by the quantitative data. However, quantitative data is best at helping you to guess when someone is happy, sad, frustrated etc., while with qualitative data, you usually get a much more reliable read over how the player is feeling, and some better suggestions on how to actually fix it.

Again, I’ve probably left some of the data you might want to consider out – this is supposed to be a formative list to give you some ideas to start with rather than an exhaustive one. If there’s anything major missing, however, I’ll be sure to update the post.

Next time, we look at how you’re gonna collect this precious, precious data: starting with the quantitative.