Focus group testing is a methodology whereby sampled individuals are brought in to experience, say, a game, and then quizzed about their reactions. It is a technique that has a long and venerable history, and yet it suffers from a severe lack of demonstrable accuracy.
The problem with focus groups is that they run afoul of a basic misconception about the way humans process information and come to logical conclusions. Jonathan Haidt in The Righteous Mind and The Happiness Hypothesis notes that we don’t have an inner scientist, but rather an inner lawyer. In other words, we are very fast to identify what we like and don’t like, but we are very poor at identifying the reasons why we like something or don’t.
What we are exceptionally good at is rationalizing and coming up with reasons post facto for why we like what we like and hate what we hate. In fact, the only time scientifically conducted studies could find people not engaging in this is when a person simply doesn’t have any context about the subject to like or dislike.
This is, in fact, the very source of confirmation bias, “the tendency of people to favor information that confirms their beliefs or hypotheses,” something endemic in politics, religion, and, well, just about everything.
Some fascinating studies of website usability in the 90s tracked people’s preferences for what browser they said they liked versus which ones they were actually faster and more efficient with, and the result was an appalling lack of statistical correlation.
How does any of this relate to gaming?
Developers are barraged daily with input and feedback. During development, this input and feedback comes from executives, Boards of Directors, leads, and paid reviewers. After a game has gone live, the input and feedback comes as much from players.
People – and this goes for executives, if they are uninvolved in the daily development process, as well as players – are very, very good at identifying what they don’t like, what is confusing, what isn’t working. Identifying the most effective and efficient solution, however, is considerably more problematic. This makes sense, as people know their own pain, but solving their pain while not causing different kinds of pain requires a depth of knowledge and awareness of production realities, financial constraints, marketing requirements, and, as well, simply different visions and preferences for a game.
What does this mean for developers?
Listening closely to the source of the pain – and happiness – of either an executive or player is incredibly important and should not be ignored. What is particularly useful is focusing on the symptoms, and using those as a starting point towards crafting a solution. For the solution, certainly take any ideas, even crazy ideas that are outside the box, and don’t be afraid to take pieces from here and pieces from there. Ultimately, the designer’s job is not to come up with the best idea, but to identify the best ideas and merge them Frankenstein-like into an amalgamated design.
What does this mean for people offering feedback on forums?
Specificity helps, and much like Robert Heinlein’s “Fair Witnesses”, focus on the particular points of pain. Where are you finding yourself stopping play? Where are you finding yourself not having fun? What are you avoiding spending your money? Then, look at the flip sides: Where are you focusing your gameplay efforts? Where are you spending your money? Remember, too, what you remember and what you do don’t always correlate (damn our own inner lawyers).
Ultimately, game design is a collaborative effort of art and functionality. It takes a vision and crafts it into a mechanic that results in engagement on the part of the player. A lack of consistent or achievable vision can kill a project’s dreams as fast as a lack of listening to the pain of a playerbase, even faster than poorly thought out or implemented mechanics. A great game needs all three of these elements – vision, mechanics, and engagement.