Submitted by Susan Griffin, Principal, Griffin + Skeggs Collaborative
My dear departed mother was conservative and devoutly religious, but rarely shocked by the falls from grace that often occurred around us. Growing up when we did, rumors about neighbors or prominent people in our social orbits, would often lead to protestations over how such a thing could happen. There would be a lot of pearl clutching in the local discourse, as well as finger pointing and, too frequently, smug condemnation.
The secrets coming to light of who fell on hard times, made questionable choices or just dumb mistakes with public consequences, led to scandal and ostracism. Often times we all knew, or suspected, the circumstances. When stories became public, the grimy details became a big deal.
Maybe it was because we had our own secrets. But when scandals broke, Mom would generally shrug, shake her head and say: “It happens in the best of families!”
She wasn’t one for “Schadenfreude” (for those who don’t know this is, it is the German word meaning “the experience of pleasure joy or self-satisfaction witnessing someone else’s troubles or humiliations”).
Mom also used to quote Bible verses like “Judge not less ye be judged” even as gossip swirled around us kids. And when the revelations of tawdry stories would emerge, she would say:” There but for the grace of God go I!”
OR my personal favorite, “People in glass houses shouldn’t throw stones!”
Perhaps it was because she knew too well . . . it could have been us in the cruel light of public scrutiny.
When contemplating recent “breaking news” about examples of survey fraud coming to light very publicly, I thought of Mom’s admonitions and their implications.
I could recount the facts, identify the parties involved, and amplify the “rubber necking” as the industry stops to gawk at the wreckage. But the reality is that the incidence of fraudulent survey data, or perhaps more charitably “unreliable data” is alarmingly more prevalent than it has ever been.
A lot about this really sucks. Not just for individual companies impacted, but for the industry at large. (Sorry for my “bad language”, but I just don’t know how to say it more elegantly.)
Fraud in survey research has been an issue for a very long time. It was a challenge in the long-ago days 20 years ago at the dawn of online research when I was in sample sales for panel provider GMI. I could tell you stories. And yes, I am that old!
Fraud is on the rise, and it is getting harder and harder to detect.
I would hazard a guess that everyone conducting survey-based research, where respondents are incentivized to participate, is deploying (or requiring their research partners to deploy) some kind of fraud detection and mitigation processes. We tick that box and many researchers go blissfully into autopilot, deferring responsibility to suppliers, even as fraudsters have gotten even more proficient in evading detection.
And really . . . in many cases we only have ourselves to blame.
Some researchers are ringing the alarm bells that survey design itself is a contributing factor: quotas with interlocking age and ethnicity factors, low incidence rates, and harder to reach groups are yielding “fewer genuine respondents”.
Again, said less elegantly, the dirty secret is that much of the survey research that gets fielded is just crap. And if we are honest with ourselves, we know this is part of the problem. Too many of the folks designing, fielding, maintaining panels of respondents, are accepting a standard of surveys that are too long, flawed in terms of their construction, deadly boring and disrespectful to the ever more endangered species of legitimate, authentic respondents.
“Faster and cheaper” also inevitably leads to “less secure” because this mantra has eliminated the opportunity and budget for effective quality control. Endorsing the “cost and speed” race to the bottom makes no sense and points a beacon to bad actors to exploit research’s vulnerabilities for their own gain.
So what is the cautionary tale for the industry?
“There but for the grace of God . . . “
NONE of us should stand in judgement.
Fraud can happen (dare I say it IS happening) to anyone doing survey-based research. Then and now.
As the industry also rushes to offer synthetic respondents to help fill hard to reach quotas, or even substitute for primary research, this real-life cautionary tale will inevitably lead to the additional question of the veracity of past data that models are being trained on.
Forget the percentage of responses that are thrown out for being detected as AI generated or real people gaming the system, even authentic respondents might not yield reliable, predictive data. Depending on how you might ask any question, genuine respondents in any target sample are vulnerable to the “say-do gap” that leads to questionable reliability of what they share. The good news is that many practitioners are reframing the discussion around the larger question of survey quality itself, not just respondent fraud.
I am no expert on these issues. The ability to trust research results has vast implications for businesses relying on them for decision support. Hopefully these challenges will be taken as a wake-up call by those who know more and are in a position to do something about them.
An important step is transparency and a true industry response that doesn’t pretend the problems aren’t real or promise quick fixes. Check out providers like EMI Research who are investing in the research-on-research that yields sober considerations around data quality in research. I am also looking forward to sessions at industry events like Greenbook’s IIEX, including brand side researchers like Brandon Olesh and Rachel Rogers from Stanley Black & Decker. They are going to share “Lessons from the Frontlines”, and how they are tackling survey fraud head-on. No magic bullets, I suspect, but candor and a dose of reality on the urgent need for attention.
As a longtime observer of the market research industry, most importantly, I can hear Mom’s words in my ears.
When it comes to survey fraud, we are all living in MRX glass houses, friends. No stones, please.
Susan Griffin, Principal
Griffin + Skeggs Collaborative
E-mail: susan@griffinandskeggs.com
