What are trap questions?
Trap questions are used in surveys as attention checks to identify respondents who aren’t answering honestly.
Survey creators give respondents a choice of answer options, where only one is correct.
These questions should be easy for a respondent to answer, as they’re only verifying that they’re paying attention. They aren’t meant to test a respondent’s knowledge.
Multiple-Choice question types are most commonly used for trap questions. These are simpler than other types and give respondents the best chance to choose the correct answer.
They’re also referred to as ‘Red Herring’ questions, Attention Checks, and Instructional Manipulation Checks (ICMs).
Trap question example
‘Lettuce’ (C) is the correct answer here. And so, you’d be able to identify respondents who aren’t paying attention. You’d have to take into account any international respondents, as not all questions will translate well.
In these cases, you could create a more obvious trap. For example:
Select D as your answer choice
- A
- B
- C
- D
- E
However, there are several risks in using traps, which we’ll address a bit further down.
Do I need a trap question in my survey?
Trap questions can be useful as a control method if you’re sharing your survey with a general audience when you can’t be sure all responses are genuine.
They’re also useful if you’re offering incentives for survey responses. In these cases, some respondents may only complete your survey for a chance at receiving the ‘prize’.
And so the trap acts as an attention check to ensure respondents are considering each question.
When buying survey respondents
If you’re purchasing survey respondents using Shout audience you won’t need a trap question, as our respondent panels are vetted and profiled for you.
Ways dishonest respondents disengage from surveys
- Primary Answer Selection: They choose the first answer listed each time.
- Positive Answer Choices (Acquiescence Response Bias): They select the positive answers for each question (e.g. ‘yes’, ‘strongly agree’).
- Neutral Answer Choices: They only pick neutral answers, such as ‘Don’t Know’, ‘N/A’, or ‘No Opinion’ for each question.
- Straight Lining: They choose the same point for every scale (e.g. answer A).
The above examples are all types of response bias, which can be a result of dishonest respondents.
How can disengagement impact your results?
The data that disengaged respondents provide are not honest or accurate. Meaning, your research findings won’t be representative of your target population.
How to add a trap question
As we mentioned above, trap questions are usually multiple-choice questions, where respondents must pick a single answer.
If you’re creating a survey with Shout, you can follow the steps below to add a trap.
- Drag and drop a Multiple Choice (Single-Select) into your survey
- Add a few answer options (4 – 6 should be enough)
- Apply Page Logic to the incorrect answers and point them to a disqualification page.
You can do this without using Logic, but the alternative is to filter your results by those who picked the wrong answer, then removing those responses.
Although, you could also create a separate report for those who didn’t fall for your trap question. This method allows you to keep all your responses.
Risks of using ‘red herring’ questions
Although these ‘red herring questions’ intend to identify disengaged or dishonest respondents, there’s also a chance they’ll cause problems in your surveys.
- Respondents may lose trust in you.
- Honest respondents may choose the wrong answer by accident
- Dishonest respondents may choose the correct answer
Damaging relationships
Regardless of whether or not the respondent is genuine, they may disengage from your survey if they realize you’re trying to trap them. As a trap question exhibits a lack of trust.
This may result in losing respondents from your survey and a reluctance to participate in any future research projects.
Losing honest respondents
Honest respondents (those who’re qualified to respond) may choose the red herring answer by accident.
And so they’ll be removed from your survey and you’ll lose valuable data.
Failed trap questions
Conversely to the above, dishonest respondents may choose the correct answer.
Meaning your trap is failed and your results are at risk of being skewed.
Best practice for trap questions
The biggest risk of using trap questions is causing respondents to lose trust and alienating them.
You can minimise this by placing a more obvious trap question at the beginning of your survey.
This will act as more of a screening question rather than as an explicit trap. This will be a signal to respondents that you are just doing your due diligence, rather than showing a lack of trust.
Answer dishonesty usually occurs because the survey is too long, too complex, or offering an incentive.
Take a look at how to conduct survey research or the survey design best practices.
