Spotting Fraudulent Survey Data with ChatGPT

There have been many excellent and thought-provoking discussions recently about survey fraud in market research which has inspired me to contribute to the conversation with a beginner’s guide on how to start spotting fradulent activity with ChatGPT.

This guide is aimed at researchers who work daily with survey data, offering practical tips on how ChatGPT can help tackle the rise in fraudulent responses effectively, even when time is limited amidst day-to-day tasks.

Fraudulent survey data can undermine insights, waste time, and erode client trust. Whether working with your own panel or third-party sample providers, quickly identifying bad data is critical.

AI tools like ChatGPT provide a scalable way to automate these checks, reducing the time needed for manual review. By building a library of tailored prompts, you can streamline your data quality checks across datasets and customise approaches for various use cases. However, AI works best when combined with human oversight to catch subtleties.

So let’s start with some basics:

1️⃣ Spotting Response Patterns

Fraudulent responses often exhibit patterns such as:

👭 Duplicate Responses: Identical entries suggest bot activity or inattentive respondents.
🚨 Speeding Through Surveys: Completion times much faster than the median may indicate disengagement.
🧮 Uniform Ratings: Straight-lining across Likert scales suggests low effort.

👉 Prompt Example: Identifying Duplicates in Large Datasets

Scan this survey dataset in Excel or Google Sheets to identify duplicate responses. Add a column titled 'Duplicate Check' and flag rows with identical responses across all fields. Summarise the count of duplicates and their frequency in a summary tab.

👉 Prompt Example: Detecting Speeding and Straight-Lining Patterns

Analyse this survey dataset to identify respondents who completed the survey in less than 30% of the average completion time or provided identical answers across Likert scale questions. Add a column titled 'Response Pattern Flags' with labels: 'Speeding,' 'Straight-Lining,' or 'None.' Highlight flagged rows for manual review.

Purpose: These approaches allow for quick detection of mechanical or rushed behaviours, enabling prioritisation of flagged entries for review.

2️⃣ Analysing Open-Ended Responses

Open-text responses are critical for depth but can reveal fraud through:

👀 Generic Answers: Responses like “good” or “N/A” add no value.
🤪 Gibberish: Random or nonsensical text often signals bots.
🧐 Inconsistencies: Shifts in tone or style suggest copy-pasting or disengagement.

👉 Prompt Example: Categorising Relevance and Tone

Evaluate open-ended responses in this survey. Add a column titled 'Response Quality' with labels: 'Relevant,' 'Generic,' 'Gibberish,' or 'Inconsistent Tone.' Use these criteria:

'Relevant' for thoughtful answers that directly address the question.
'Generic' for responses like 'okay' or 'good.'
'Gibberish' for nonsensical or random text.

'Inconsistent Tone' for abrupt shifts in language style or complexity. Summarise flagged responses in a new tab for review.

👉 Prompt Example: Highlighting Repeated or Stock Answers

Analyse open-ended responses to detect repeated or stock phrases used across multiple entries. Add a column titled 'Repetition Flag' to indicate whether an answer appears more than three times across the dataset. Summarise repetitive phrases and their frequency in a separate tab.

Purpose: These approaches focus on both individual response quality and trends across open-text fields, helping isolate low-quality or fraudulent entries.

3️⃣ Checking Logical Consistency

Fraudulent or careless respondents often provide answers that fail basic logic checks:

🧍 Demographic Contradictions: E.g., age doesn’t align with years of work experience.
🤔 Conflicting Answers: Responses in related questions don’t match up.

👉 Prompt Example: Detecting Demographic Errors

Check for inconsistencies in demographic data within this dataset. Add a column titled 'Demographic Check' and flag rows where:

Age is under 18 but employment is over 10 years.
Age and marital status conflict (e.g., under 18 and married).* Provide a summary of flagged inconsistencies.

👉 Prompt Example: Identifying Contradictory Responses in Related Questions

Cross-check responses to related survey questions. Add a column titled 'Logic Check' to flag:

Respondents who rate a product poorly overall but highly rate its specific features.

Contradictory answers in preference or behavioural questions (e.g., stating they dislike an activity but indicating frequent participation). Summarise flagged rows with explanations in a separate tab.

Purpose: These prompts address logic gaps across demographic and behavioural data, ensuring consistency in responses.

4️⃣ Analysing Metadata for Red Flags

Metadata provides valuable clues about potential fraud:

💻 IP Address Clustering: Responses from the same IP may indicate bots or shared devices.
📱 Device Consistency: Identical device/browser configurations across entries suggest automation.
🕑 Completion Time Outliers: Responses far outside normal durations could indicate low-quality data.

👉 Prompt Example: Identifying IP and Device Clusters

Analyse metadata to detect repeated IP addresses or identical device/browser configurations. Add a column titled 'Metadata Flags' with labels: 'Repeated IP,' 'Identical Device,' or 'None.' Summarise flagged metadata patterns and the associated response IDs in a new tab.

👉 Prompt Example: Highlighting Completion Time Outliers

Analyse survey completion times and identify outliers below 25% or above 150% of the average. Add a column titled 'Time Check' with labels: 'Fast,' 'Slow,' or 'Normal.' Highlight flagged entries for manual review and summarise average completion times in a separate tab.

Purpose: These prompts focus on using metadata to uncover suspicious response patterns and anomalies.

5️⃣ Detecting Outliers

Outliers can signal fraud or low-quality data:

🔍 Behavioural Outliers: Extreme scores on unrelated questions.
📊 Demographic Outliers: Unusual combinations that don’t align with the target audience.

👉 Prompt Example: Flagging Extreme Scores Across Variables

Identify responses with scores more than 2 standard deviations above or below the mean for any variable. Add a column titled 'Outlier Flag' and label responses as 'High Outlier,' 'Low Outlier,' or 'Normal.' Summarise flagged data in a new tab.

👉 Prompt Example: Identifying Unusual Demographic Combinations

Analyse demographic data to detect unusual combinations (e.g., age under 18 with high income). Add a column titled 'Demographic Outliers' to flag these entries. Provide a summary of flagged patterns and their frequency in a separate tab.

Purpose: These approaches help researchers isolate responses that deviate significantly from expected norms.

6️⃣ Classifying Responses for Review

To streamline workflows, categorise responses based on overall quality:

⬆️ High Quality: Thoughtful and consistent answers.
⬇️ Low Effort: Straight-lined, rushed, or generic responses.
🚨 Suspicious: Entries requiring further review.

👉 Prompt Example: Creating a Quality Categorisation System

Classify survey responses into High Quality, Low Effort, or Suspicious categories. Add a column titled 'Response Classification' with criteria:

'High Quality' for consistent, relevant, and thoughtful answers.
'Low Effort' for straight-lined, generic, or nonsensical answers.

'Suspicious' for entries with metadata flags or logical inconsistencies. Summarise flagged responses and their categorisation in a new tab.

👉 Prompt Example: Combining Metadata and Response Patterns for Classification

Integrate metadata flags, response patterns, and open-text analysis to classify responses. Add a column titled 'Integrated Quality' with labels: 'High Quality,' 'Low Quality,' or 'Flagged for Review.' Highlight patterns across categories and provide a summary of flagged trends in a separate tab.

Purpose: These prompts provide a holistic view of response quality, integrating multiple checks into a single output.

7️⃣ Building a Master Prompt

Market researchers can consolidate these approaches into a master prompt for streamlined analysis:

Analyse this survey dataset for fraudulent or low-quality responses. Perform the following checks:

Identify duplicates, straight-lining, and speeding patterns.
Evaluate open-text responses for relevance, gibberish, and tone consistency.
Check for demographic and logical inconsistencies.

Analyse metadata for repeated IPs, device clustering, and completion time outliers. Add columns for 'Response Quality,' 'Metadata Flags,' and 'Consistency Check' in Excel or Google Sheets. Categorise responses as High Quality, Low Effort, or Suspicious. Provide a summary of flagged entries in a separate tab.

FInal Thought

While AI offers efficiency and scalability, human expertise is essential to spot nuances AI may miss, such as cultural context or subtle inconsistencies. Use AI for initial checks and prioritisation, but rely on your experience for final decisions.