Rethinking Attention Checks for Stronger Data Integrity

by Ellyn Steidl

‍For consumer insights to be actionable, clients need to be able to trust data quality. Best practices for online survey questionnaires involve multi-layered approaches to quality control throughout the research process. This can include automated technological strategies during fielding and hands-on methods of review post-data collection. One way to promote data integrity prior to fielding is to add attention checks into a survey.

Attention checks are questions that gauge a survey taker’s attentiveness and engagement. They help identify respondents who aren’t invested or whose answers could end up skewing study results. Traditional attention checks can include factual questions (e.g., “What is 5+5?”), text-based questions (e.g., “Please type the word ‘orange’”), direction-based questions (e.g., “Please select the option ‘Somewhat Agree’”) and speed-monitoring, which flags respondents who race through the survey. Although these types of attention checks are commonly used, they may not be all that effective at catching survey takers who give low-quality responses.

A newer approach to attentiveness tackles the issue directly. Researchers at Qualtrics tested traditional attention checks alongside what they termed a commitment request, that is, a question asking respondents if they “commit to providing thoughtful answers.”  

Respondents were randomly assigned to an attention check group before completing the survey. Then, the researchers analyzed whether survey takers went on to fail two or more other quality indicators, such as low-quality open-end responses or always selecting the same answer in a question grid (i.e., straightlining). Interestingly, respondents who made a quality commitment had the lowest failure rates of all groups. In other words, survey takers who promised to provide thoughtful answers were indeed more likely to provide high-quality responses.

Why is that? Commitment requests may be more effective because they establish a set of values for survey takers by asking them to opt into integrity. Addressing the need for quality data and explaining why thoughtful answers matter can establish a shared purpose that sparks deeper engagement. Requiring survey takers to go “on record” with their intentions can also create a sense of responsibility for their responses.

Commitment requests may also reassure respondents that their feedback is valuable. Survey audiences want to know their answers matter and are taken seriously. Including a data integrity message at the beginning of a survey can help signal this commitment on behalf of researchers. After all, as researchers we want people to share their honest opinions with us, so sometimes a simple reminder can nudge people in the right direction.

Commitment requests should operate alongside additional quality control measures. Here at W5, consultants use a combination of automated and hands-on quality control techniques while always learning new strategies.

If you need insights and feedback from consumers, professionals, patients, caregivers, B2B decision makers and more, the W5 team is ready and able to partner with you on your next project.