Focus Group Dystopia: Marketers Beware

I've been arguing against the validity of focus groups for some time now and I'm certainly not alone.  They are not effective tools for innovation, insight generation or validation. But now they are outright dangerous.

Focus groups lend themselves to being gamed.  We saw it very early in the dot com boom where the same people would show up regardless of how you worded the screener.  It didn't matter which firm you used, or whether you wanted someone with 1 year of experience online, 3 years of experience or a PhD in computer science.  Marla and Jim would be in the submitted names.  They were professional focus groupers and had registered with every research house in town.

So the market reacted and higher end houses worked to identify and screen these people out.  But now a new variation appears to be at play.  One where cheating the marketers is not only the order of the day, it's a business.

A friend of mine loves focus groups.  Let's call him Dan.  As an actor, focus groups allow him to make some easy cash between gigs.  The other day I met Dan for lunch and he informed me that he had a focus group to attend after our meal.  Nothing unusual.

What was unusual was he had notes.  While we were waiting for the bill he was studying them quite intently.  I was quite perplexed and asked him what the notes were for.  He smiled awkwardly and said he had to be a gay man between jobs with two kids from a previous divorce.  He was trying to memorize his previous work history and his views on dating sites.  My jaw hit the table.

His recruiter didn't call with a screener.  His recruiter called with briefing notes on who he was to "become" for the session.

I pity the marketer still using focus groups.  It's only going to get worse.

Image Source: Oliver Ingrouille

Comments

[Taylor Davidson]

Focus groups are great for gauging what people say in response to questions that have no impacts, without having to make a commitment or invest any time or money into something, knowing that whoever they are with (other participants or just the moderator) is judging them, their "performance", their intelligence, their helpfulness.

They tell you very little about what people will actually do.

[Sean Howard]

Hey Taylor,

So if a marketer wants to know what people will say in an environment that has no impact, where the participants know they are being judged and feel pressure to be helpful... then focus groups are a great fit? I can't agree more.

I often wonder why we pay for focus groups and why, if we want to test an idea, we don't just express the idea and let people respond anonymously. Like on an ideas blog like a bunch of brands have launched of late, for example.

[Mike Wagner]

Does our emerging "game savvy" culture result in a natural instinct to "game" a lot of what marketers do to learn about "real people"?

Or is it just pay back for having felt manipulated by marketers? Grin.

Thanks for stirring the pot Sean!

Keep creating...a story worth repeating,
Mike

[Taylor Davidson]

@Sean We pay for focus groups because it gives use the illusion of agreements.

It's also very hard for most people to truly sit in someone else's shoes.

The problem with the "letting people respond" method of testing is that it still suffers a couple problems; one obvious one is the self-selection bias of the people that respond, which can lead to some pretty fundamental misreads on what people overall will actually do.

@Mike I think we've always been skeptical of people who are trying to sell us something or get something from us. It's just more obvious now.

[Sean Howard]

@Mike: Stated in jest but dead on. I have seen this desire to "screw the marketers" talked about on more than one site/area. To date, there hasn't been the ability for people to target and coordinate such an initiative on a scale large enough to be of serious concern for most marketing research initiatives. But that is no longer the case...

@Taylor: Indeed and agreed re: bias. But consider the sample size for a focus group. Likely worse than the hundreds or thousands of people that participate in brand idea sites. But you raise an interesting point. No reason we can't request and parse the data post idea. Start to understand who these people are and what the bias might be in the data.

[Kelly]

I'm one of those qualitative researchers wasting everyone’s money. Watch your cash – I’m going to BURN it like it’s going out of fashion!!

Stories like this Sean make me feel a little ill; but as a moderator with a bit of experience you can normally sniff people out who have created their “back-story” pretty quickly, and you begin to work out as well the recruitment companies that brief their respondents in this way, and stop working with them. We have “groupies” here who also like to attend groups to get some cash, eat some sandwiches and while away an hour or so.

I guess the big issue I have with your post is the “danger” implied in using a focus group, or the lack of value it will deliver. The danger for me is not the focus group itself, but the type of marketer or researcher you are. The marketer who uses verbatim responses in a group to justify a strategy is the biggest threat to sound decision making. (“But in the group all of them said they’d buy it, all of the time, and pay a $100 for it!”). This rubber stamping approach is the lazy and dangerous use of focus group data. Or the researcher who asks straight out “hands up who thinks this is a great idea?” This is what gives qualitative research a bad name.

I have read some research reports where you think to yourself “Yep, the client just pissed $80K up the wall for that project”. But there have been other qualitative projects (using any number of qual methods including the focus group) that have illuminated the way consumers think, feel and act about a category or a brand that has delivered absolute value. So rather than dismissing all qual research as a waste of cash, it’s probably where, who and how you chose to spend your cash, just like with any service provider.

I think there is also real disparity with the quality of output in qualitative research between North America and the UK/ Europe mainly based on the methods used (and hence also used here in Aus and NZ) and this also plays a big role out of the value derived from qual. From what I understand and have seen (and I may be grossly exaggerating, but is based on client feedback so excuse my Antipodean ignorance) there is a real difference in the types of questioning and techniques used within groups, big differences in the analysis process, and a BIG difference in how results are then used within business. And again, this is a generalisation, as I know there are a lot of crap researchers here and in the UK, so certainly not saying “we are better” (or maybe I am…)

[Sean Howard]

@Kelly:

Thank you for joining in the conversation. I certainly did not intend to malign qualitative research as I believe strongly in it. I did set forth to put down focus groups but certainly could not argue that all focus groups are the same. And interestingly, there are likely cases where I would use something very akin to a focus group.

In stirring up the pot, I avoided defining what I was attacking and thus lumped in all forms of focus groups regardless of their value. I wish for the industry that more people like yourself begin to set the tone and manner of how focus groups are approached and used as I've sat through a strong majority of horrific ones. Which again is a statement against the firms I worked for and not yourself.

[Taylor Davidson]

@Kelly: Sean summarizes my response spot on. I've sat through bad and good focus groups and bad and good focus group moderators.

The real problem is the type of decisions many companies commonly try to answer with focus groups: not a problem with qualitative research but with how companies use the data to structure decisions.

[Simon K]

One of the issues I've found with product or concept testing in focus groups is from the client end rather than the participant. I wonder how many focus groups are completely balanced and unbiased, with the results in no way skewed to reflect the particular pre-existing preferences of the client...

[Asian Gay Dating]

cheating never works in anything... might work for a moment, but it wont last!!
your post is really nice and I really enjoyed reading it!