Every survey goes through a set of behind-the-scenes checks the moment a respondent hits submit. Those checks determine whether the results can be trusted.
Invalid: a survey that didn't produce a scored result because the respondent either wasn't paying attention, responded in a pattern that didn't seem authentic, or otherwise engaged with the survey in a way that seems incorrect or unrealistic.
Not every flag makes a survey invalid. Some are advisory. Here's how the flags break down.
Flagged but still valid
These flags leave the survey valid, meaning the results still score and show up in the dashboard, but they're worth a second look and a retake is recommended.
Elevated Social Desirability
"This respondent may be managing their impression"
Social Desirability Responding (SDR) is when a respondent answers in a way that makes them look overly good. Endorsing every positive adjective and rejecting every negative one, past the point of realism. Saying Charismatic is Like Me and Nervous is Not Like Me is fine individually, but a whole survey of that pattern isn't credible.
This flag means the respondent landed close to the SDR threshold but didn't cross it. Results are still valid, but the respondent is permitted to retake, and it's recommended they do.
Timing
"The respondent took too long to complete the survey"
Each survey has a time allowance. Personality, for example, has a 30-minute timer. If a respondent takes longer than that, we can't be sure they were still engaged with the questions. Results are still valid, but retake is recommended.
Invalidating flags
These flags invalidate the survey. No scored results are produced, and the respondent should retake.
Attention checks
"This respondent failed attention checks"
The survey mixes in two kinds of attention checks. The obvious ones are direct instructions like "Select Like Me," where the respondent literally has to follow the instruction. The softer ones look for contradictions between opposite adjectives. If a respondent misses too many of either kind, the survey is invalid.
Strong Social Desirability
"This respondent is strongly managing their impression"
The respondent went past the allowed SDR threshold of favorable-direction answers. See Elevated Social Desirability above for what SDR means.
Signs of manipulation
"This respondent appears to be manipulating the survey"
If a respondent looks like they were trying to figure out how the survey works or get around it, we flag it. That includes things like jumping between browser tabs a lot, taking screenshots, or opening developer tools. When enough of that behavior shows up on a single response, the survey is invalid.
Signs of automation
"This respondent appears robotic or responded unrealistically"
The system also watches things like typing rhythm and mouse movement. When those look robotic or too perfect, in patterns a real person's hand wouldn't produce, the survey is invalid. This usually catches bots.