Her Story creator Sam Barlow explains why focus groups can harm games amid Wolverine criticism

Immortality and Her Story creator Sam Barlow spoke out about problems with focus testing in movies and games – and how it can end up shaping creative decisions for the worse. The trigger was recent design changes made to Marvel's Wolverine, and Barlow put it bluntly: this is what happens when all the money goes to focus testing and none of it to consultants.

According to him, movie focus tests almost always end up boosting requests for more clarity, more explanation, and more exposition. Games get a similar result: testing tends to leave the impression that players are confused, frustrated, and lost.

The issue is that figuring things out on your own is part of the joy of both mediums. If a viewer walks away from a movie still wondering whether one character truly loved another, or what happened to a third one, that is not necessarily a flaw. That lingering question can keep working on someone long after the credits roll.

Barlow argues it is far more useful to run tests built around one clear question and then watch how players actually behave, rather than asking them to rate the game as a whole. For example: "Do players realize how useful the secondary fire mode is?" From there, you can track how often and when players actually use that mechanic over several hours of play.

Questions like "Did you have fun?" or asking someone to explain why they played the way they did tend to produce far less reliable answers.

Barlow compared this to a chef who only accepts "I'm stuffed!" as a good review, then panics when someone says "that was so delicious, I want more," reading it as a complaint that the portion was too small.