Are Bots Ruining Your Surveys? Here’s How We Catch Them!
October 2024, by Hayley Kingston
At MM-Eye, our mission is to help our clients make informed business decisions by investigating what really matters to their customers, employees and stakeholders. We rely heavily on the quality of the data we collect to ensure that we can provide tangible, accurate insights for our clients.
One of the biggest challenges faced by the market research industry at the moment is ensuring the quality of the data we collect. We see fraudulent respondents undertaking surveys and developing technologies, including AI and Language Models, that are used by bots and click farms, meaning that we need to be even more vigilant and stringent with our quality control checks.
To ensure the data we collect is free from sub-par or fraudulent responses, we have established various techniques for “weeding out” these responses at different stages of the data collection process. Firstly, at the survey completion stage, we can implement red herring questions in our surveys, designed to catch respondents who are not paying attention to ensure that respondents are actively reading the questions. In addition to these, we can implement straightlining checks to flag when respondents select the same answer for most or all questions within a survey, as this indicates that they may not be providing attentive responses.
Looking at respondent behaviour is another way we can ensure the reliability of our data during data collection. Several behavioural indicators help us identify whether respondents are providing thoughtful and accurate answers or simply rushing through surveys. For example, the time it takes for a respondent to complete a survey is a strong indicator of response quality. We have an idea of how long a survey should take to complete, so respondents who complete too quickly may not be paying attention or completing the survey appropriately. Therefore, we implement speeder checks in our surveys, screening out responses that fall below a reasonable threshold of completion time.
As well as filtering out low-quality responses within the survey completion, we conduct thorough and ongoing checks throughout the fieldwork period on our open-ended questions. These checks are an essential part of our process to ensure that our data is clean, accurate, and reliable.
Every day, our team manually reviews all open-ended verbatim responses to identify fraudulent responses, bots, or individuals who have not taken the survey seriously. Some of the things we look for are gibberish/keyboard spam, nonsensical answers that do not answer the question we’re asking, and answers that have been copied from the internet or generated by an AI model.
In addition to checking open-ended responses daily, we perform a comprehensive review of the entire dataset weekly. This includes a deep dive into both open-ended and closed-ended questions, using our years of experience to identify broader patterns that could indicate data issues or whether bots/click farm respondents are present.
Maintaining data quality is an ongoing and ever-changing process that requires a combination of preventative measures, real-time checks, and regular reviews. By monitoring respondent behaviour, implementing tools to weed out poor quality responses, and conducting daily and weekly checks, we ensure that the data we collect is both reliable, trustworthy and accurate.
Our goal at MM-Eye is to deliver actionable insights that inform critical business decisions, and these rigorous processes ensure that we achieve that goal with data that we and our clients can trust.
If you’re interested in learning more about our data quality solutions and how they can benefit your business, contact us today at info@mm-eye.com or use the form below to book a consultation.