Surveys seem easy to write.
However, there are uncountable variables to account for with survey design to ensure the answers you receive are accurate, inclusive, and honest.
Any of which you need to correct can throw off multi-million dollar decisions made around operations, marketing, or strategy.
Unfortunately, many organizations that utilize surveys do not test them properly. Whether the market research is conducted internally or externally with the target respondents, proper testing is only taking place in some cases.
Cognitive survey testing solves that.
It ensures your market research study is well set up for success, and you can have complete confidence and faith that the answers you receive from your sample audience are 100% on point.
What Are Cognitive Testing Surveys?
Cognitive testing surveys are used to evaluate how well the intended research respondents understand and answer questions in a market research questionnaire.
It might include asking about behaviors, purchase patterns, opinions on brands, likes and dislikes of an advertisement, or drivers to loyalty or satisfaction.
Cognitive testing is a much more involved process of simply pretesting a survey. Since most quantitative surveys are taken online via mobile devices, there is a massive disconnect between the survey writer and the respondent.
With most surveys being taken online, the survey writer often needs feedback on interpretation, points of confusion, the reasoning behind decisions, etc., without doing live pretests or cognitive testing.
Cognitive testing focuses on assessing respondent processes and what they think when forming thoughts or making decisions in a survey setting. When running cognitive tests with a live facilitator or moderator, respondents are often encouraged to think out loud when answering survey questions.
Cognitive Survey Testing Example
The survey writer often analyzes the data received but is far removed from the respondent experience when the person answers their questions live on the couch, at their desk, on the bus using their mobile device or watching their favorite TV show on their tablet.
Whereas, cognitive testing helps researchers get inside the respondent's mind when answering their questionnaire before it is launched to the masses.
For example, a survey question may read: How often do you buy dog food?
- Once a week or more often
- Once every 2-3 weeks
- Once a month
- Once every 2-3 months
- Less often than once every 3 months
- I do not purchase dog food
Without cognitive testing, you may get the results back and learn that 75% of consumers purchase dog food once every 2-3 months.
So, assumptions are made that the dog food consumer is buying those items in bulk to last them a long time. Therefore, marketing and strategy are designed to cater to those who purchase large 40+ lb. bags of dog food since those messages likely resonate with 3 quarters of the audience.
However, through cognitive testing, you might learn a different story. 📖
Although respondents purchase every 2-3 months, they aren’t buying large bags or bulk dog food sizes. You realize that out of those 75% of consumers, more than half purchase many smaller bags of dog food in bunches.
- Respondent A thinks out loud to the moderator and shares he waits for their 5-10 lb. bags of food to go on sale at the local Walmart and then purchases 5-10 of those smaller bags at once, which ends up being much cheaper than buying the bulk 40 lb. bags.
- Respondent B walks the facilitator through this question and says she buys her dog food through Amazon. There is a prime deal that hits every 2-3 months where she can buy a multi-pack of cans of soft dog food every 2 months or so where she is purchasing 8-10 12-packs of cans for her puppy. It also becomes more cost-effective than buying the larger bulk slate of cans.
Without cognitive testing, survey results can lead you down the wrong path. Conversations like these with actual respondents help you formalize the survey script more accurately to account for sizes of dog food purchases, how they are purchasing, factor(s) in choice for purchase, type of dog food purchased, etc.
Why is Cognitive Testing Helpful in Market Research?
The goal of cognitive survey testing is to understand how target audiences perceive and interpret the questions asked in your survey, which enables the research team to create better and more appropriate questions.
Garbage in, garbage out, as they say, regarding wrong questions and bad data. 🗑️
Here are the ways cognitive testing will help you for your market research project:
Ensures clarity and comprehension
Cognitive survey testing helps ensure your research materials, such as screeners, questionnaires, surveys, or instructions, are easy to understand and provide clear asks. This is crucial for accurate data collection and minimizing misinterpretations. If a respondent needs to interpret questions, it leads to falsified data.
For example: Typically, how many ounces of coffee do you drink each day? If respondents need help to convert cups to ounces or miscalculate ounces, it creates significant discrepancies in the data to analyze. Cognitive testing can help standardize a measurement or identify confusion.
Reduces response bias
Mainly, if you are asking about a complex topic, a new product or service, or a subject with acronyms, all could be a source of confusion with respondents, leading to bias. Pre-testing the survey through cognitive testing can eliminate this and create more strongly worded questions.
For example: How would you rate our customer service and prices? It is a double-barreled question. Through cognitive testing, you may learn some respondents rate this a 3 on the 1-5 scale because they viewed customer service as a 5 and prices as a 1 and averaged that to a 3. You determine it’s better suited as 2 separate questions.
Improves survey design
Through cognitive pretesting, researchers can refine the design of their questions, ensuring they are phrased correctly.
It is especially critical for international studies or diverse populations where the survey may be translated into different languages. Topics are framed differently, and foreign words are used depending on the country where your research is focused.
Cognitive testing ensures your questions are culturally sensitive. If your survey is being administered in multiple countries and languages, cognitive testing is a must to ensure it is culturally appropriate in all aspects.
It is crucial since there are so many unknowns in different countries. Without understanding issues directly from respondents, it can create problems.
Validates measurement scales
Cognitive pretesting can help validate measurement scales used in research.
Whether you are testing usage quantities for consumer behavior or price research, it is a must-do. ROI in cognitive pretesting can be as simple as accurately using unit conversions, currencies, etc.
Crucial in market research.
Saves time and resources
Detecting and fixing issues with surveys before the launch will save you time from trying to understand mishaps or discrepancies in data that may not be easily understood on the surface level of analysis.
Re-fielding a survey is no fun for anyone. Well-validated surveys create better ROI.
Although cognitive testing is centered on big-picture comprehension and understanding, there’s nothing wrong with getting a little value by identifying typos, grammar issues, etc., either!
Increases respondent engagement
When respondents find research materials easy to understand and relevant to their experiences, they are more likely to cooperate and provide thoughtful responses.
It can improve response rates and data quality with a more extensive set of respondents who enjoy the experience, take the questions to heart, and think through their answers. Engagement is enormous in market research. If the experience is enjoyable, it will translate to better data quality.
ROI of Cognitive Testing Surveys
Cognitive testing is often a skipped part of the process in market research but should be considered a best practice. The question you should ask yourself is:
“Do you have the additional budget and time to ensure you get the market research right?”
If the answer is no, cognitive testing is not a fit for your quantitative survey project.
However, you risk respondents misinterpreting questions, answering incorrectly, and potentially leading your research team down the wrong path with decisions based on your research. It’s the higher-risk path.
The ROI of cognitive testing with surveys is nearly a guarantee.
No matter how many cognitive tests you run (1 or 10), you will discover insights and learn about items that should be updated, changed, or better explained in your survey.
The lower risk path is saying yes to that question and proactively building in the time to run cognitive tests with potential participants.
It reduces the risk and is extremely valuable for high-leverage and critical studies at your organization (e.g., launching a new multi-million dollar product, creating a new $1M advertising campaign, etc.).
You must make sure everything runs smoothly. But don’t forget cognitive testing for smaller-scale studies, either. Even a handful of interviews can help make valuable changes to your questionnaire.
How to Conduct a Cognitive Testing Survey
A top cognitive testing market research company will have a systematic approach to the project.
Not only will that organization help recruit individuals to participate in the tests, but they will also create a deliverable that will improve your survey by leaps and bounds.
If they are a full-service firm, they will likely be able to help you with the entire data collection process post-cognitive testing, including any analysis, insights, or recommendations on the outcomes of your survey.
Whereas cognitive testing focuses on the instrument, the same firm can also help with the analysis and interpretation of the data.
Here are the steps to conduct a cognitive testing survey:
- The first step is understanding your research objectives, goals, and target audience.
- From there, develop your survey draft to address all primary and secondary objectives.
- The draft survey becomes your pretest survey to utilize for cognitive testing.
- Recruit participants to take your survey and schedule a dedicated date and time.
- The facilitator/moderator joins the meeting to probe and ask questions to the respondent.
- If it is an online survey, there is typically a screen share where the respondent thinks out loud.
- The moderator identifies hesitation, confusion, delay, etc., and probes for explanation.
- This process is followed by a dive deep into the intended ask for each question.
- Probing is a requirement, diving into difficulty in understanding confusing and ambiguous questions.
- Pay attention to response options and if the answers are mutually exclusive and collectively exhaustive.
- Observe patterns in answers or note inconsistencies or contradictions in responses.
- From there, modify and refine the survey based on the feedback to create a new version.
- Conduct additional rounds of cognitive testing in other languages if necessary.
- If significant changes were made due to a complex topic, consider Wave 2 of cognitive tests.
- Pre-launch the survey to ensure changes made have an impact (e.g., 5 or 10% of the total sample).
- Proceed to a full launch of your survey and continue to monitor discrepancies, drop-offs, etc.
The most common and suggested methodology for cognitive survey testing is one-on-one interviews.
Since it removes group bias through individualized conversations, it allows the researcher(s) to gain a deeper understanding of how each consumer processes information and makes decisions in response to survey questions.
The insights obtained from cognitive testing will undoubtedly create a more refined and accurate survey. It is better to run cognitive tests among 5-15 participants before your online survey launch to ensure the 400, 1000, or 2000+ responses are accurate, reliable, and high-quality.
Cognitive Testing Survey Example Questions
Ultimately, working with a cognitive testing company will ensure the questions asked as part of the probing during the interview are customized and unique to your project. However, some basic high-level questions can be good examples for any project.
- In your own words, what are we asking you here on this specific question?
- For this question, how did you decide on this answer? Can you walk me through it?
- Specific to this phrase used, what do you think it is asking? Is that the correct phrase?
- You rated this product a “5” on satisfaction. Can you explain why? What goes into the “5” rating?
- You hesitated. Does this question apply to you, or did you feel forced to select a response?
- You spent a lot of time here. Was this particular question easy to answer or not?
- Were we missing any answer categories here? Were the answers exhaustive?
- How confident were you with that answer? Why or why not?
- How could these instructions be stated more clearly?
- You answered that question quickly. Did you understand that acronyms?
Other Tips for Bias-Free Market Research
Conducting bias-free market research is crucial to ensure that the data and insights collected accurately represent the target population and can be used effectively for decision-making.
If your market research contains any type of bias, it can send your team down the completely wrong path. Bias is the evil part of market research that needs to be eliminated as much as possible.
Here are some tips to help you minimize biases in your market research:
Random sampling
When selecting your research sample, use random sampling techniques to ensure that every member of the target audience has an equal chance of being included. So much can be said for random sampling as it is the great equalizer.
When you take a random sample of a larger list without any prejudice, it ensures the sample of survey takers is representative of the intended population.
By using random number generators in Excel and pulling 5,000 contacts out of a list of 50,000 - there is a strong chance the 5,000 will mimic the make-up of the 50,000.
Avoid leading questions
Craft survey questions that are neutral and do not lead respondents toward a particular answer.
An example of a leading question: “Our customer service was rated in the top 5 in our industry for the past 4 years. How would you rate our customer service?”
It creates biased and forced responses. The idea of a market research survey is to be objective.
You need to know the good and what to double down on in marketing and strategy, but you also need to know the bad to improve. Creating bias that steers a respondent is not a best practice or recommended approach.
Soft-launch your survey after cognitive testing
Conduct pilot tests to identify and rectify issues related to question wording, response options, or the overall survey design that might introduce response bias.
Even after you complete your cognitive tests, start your fieldwork slowly.
We recommend sending invites to 5-10% of your intended sample and scrutinizing those responses to see if further adjustments are necessary before a full launch.
Recommended Reading: 7 Reasons to Soft Launch Online Surveys
Minimize acquiescence bias
This occurs when respondents are overly favorable to agree with a concept, service, etc. Make sure you balance this by including positive and negative terms and appropriately balancing any Likert scales.
It is a phenomenon in market research where respondents aim to please, thinking it potentially has some impact on being chosen for the study, a higher reward, or qualifying for the next level.
It is essential to create balance in your survey and even mention in the cognitive interview we want to hear the good and the bad.
Recommended Reading: Acquiescence Bias in Market Research
Confidentiality
Ensure participants their responses will be anonymous and confidential. It helps ensure honest and forthcoming answers.
If the respondent feels as though they will be ridiculed or receive a follow-up phone call directly after submitting negative feedback, it will likely prevent them from responding at all.
Contact Our Cognitive Testing Survey Company
Drive Research and our team of research experts to help guide you through the cognitive testing of your survey.
We manage the whole process from end to end, including recruitment, moderating/facilitating, and summarizing the feedback to create a valuable and impactful report driving improvements to your survey instruments.
Our research firm has the capabilities to translate internationally and work with local partners in each country to create and moderate.
As a full-service market research company, we can also work with you on a post-cognitive test to collect responses online and develop insights from your survey to drive your operations, marketing, or strategy.
- Message us on our website
- Email us at [email protected]
- Call us at 888-725-DATA
- Text us at 315-303-2040
George Kuhn
George is the Owner and President of Drive Research. He has consulted for hundreds of regional, national, and global organizations over the past 15 years. He is a CX-certified VoC professional with a focus on innovation and new product management.
Learn more about George, here.
