Not all synthetic answers are easy to spot.
In a previous post on fraud detection, we covered the basics: speeders, straight-liners, generic phrasing.
But as more respondents turn to AI tools to “help” with surveys, our detection strategies must get smarter and more precise.
So let’s take a closer look at how we might tackle this, using a series of example prompts. Some of these ideas still need refining and critical testing, but having trialled a few on my own datasets, they’ve proven surprisingly effective at detecting anomalies and inconsistencies in the data.
🧠 1. Reverse Inference: Can You Reconstruct the Question?
AI-generated answers often respond directly to prompts without true interpretation. Humans reflect or reinterpret. Prompt backwards.
👉 Prompt: Reverse Relevance Check
“For each open-ended response, infer the likely original survey question. Rate the plausibility of the inferred question using a Likert scale from 1 (Not Plausible) to 5 (Highly Plausible). Flag responses rated 1 or 2, where the original prompt is unclear or could not be inferred.”
📌 Why this matters: If the answer is vague or over-generalised, it likely wasn’t written with real context in mind.
🧬 2. Style Drift: Cross-Response Consistency Within a Respondent
Real respondents have a writing voice. AI tends to generate in fragments, especially if prompted for each question individually.
👉 Prompt: Internal Consistency Scoring
“Compare sentence structure, word frequency, sentiment range, and syntactic complexity across all open-text answers from each respondent. Assign a ‘Style Consistency Score’ from 0 (No Consistency) to 1 (Perfect Consistency). Flag entries below 0.5 for manual review.”
📌 Why this matters: AI-generated entries stitched together often lack consistent voice.
🎭 3. Emotional Contradiction Detection
AI can mimic emotion, but struggles to maintain tone across multiple answers.
👉 Prompt: Emotional Consistency Check
“Analyse the emotional tone of each open-text response per respondent. Use emotion tags (e.g. Joy, Disgust, Frustration, Pride). Flag inconsistencies where tone shifts drastically between related questions, such as negative tone in a satisfaction rating but positive tone in the follow-up explanation.”
📌 Why this matters: Emotional drift suggests the response wasn’t anchored in personal experience.
🛰 4. Metadata and Semantic Layer Fusion
Take Part 1’s metadata checks further by layering in language similarity.
👉 Prompt: Multimodal Pattern Recognition
“Create clusters of responses with shared metadata (same IP, browser, device, time pattern). For each cluster, analyse semantic similarity using cosine similarity of response vectors. Highlight clusters where both metadata and semantic similarity exceed 0.9.”
📌 Why this matters: If both what and how someone responds looks identical, you’re likely seeing automation.
🕳 5. Narrative Holes and Overcommitment
AI often inserts unnecessary detail, switches tense, or breaks logic.
👉 Prompt: Plausibility and Narrative Logic Scan
“Evaluate each open-ended response for narrative logic. Highlight responses with over-specified detail that lacks context, abrupt tense changes, or illogical statements (e.g. ‘I always loved this product even though I just discovered it last week’). Tag responses as ‘Plausible’, ‘Mildly Illogical’, or ‘Contradictory’. Summarise contradictions per respondent.”
📌 Why this matters: Humans rarely contradict themselves like this. AI often does.
🔄 6. Dynamic Rephrasing to Test Fragility
Real responses hold up when you rephrase the question. AI responses often don’t.
👉 Prompt: Fragility Check via Rephrased Questions
“Take each open-ended response and attempt to rephrase the original question in two or three different ways. Then prompt ChatGPT to regenerate a response for each rephrasing. Compare the generated outputs to the original human response. Flag discrepancies where the original does not align semantically with the variations.”
📌 Why this matters: AI-written answers are often overfitted to the original question.
🔐 Bonus Strategy: Embed Contextual Traps
If you’re designing the survey, create consistency checks.
- Ask the same question in different ways
- Insert a memory cue early and reference it later (e.g. “Remember this code: BLUE47”)
- Include a fake brand or feature and test for false agreement
📌 Why this matters: People forget. AI fabricates. Both reveal different things.
Final Musings >> For Now 🧠
We’re now verifying not just who answered your survey, but how they answered it.
Authenticity is no longer assumed. It’s something you have to build into the process.
Stop thinking of AI as just a fraud detector. Use it to rethink survey design, question formats, and response validation.
