As our Vice President of Strategy recently stated on social media, market research is easy if you know what you're doing, but there's a good chance you have no idea what you're doing.
Or something along those lines.
He stated you could mow your lawn, replace your roof, or paint your house, but if you use a professional, chances are they'll get it done better in less time and it will last longer.
Unfortunately, there are far too many examples of bad market research that come to mind.
Here’s what happens.
Brands start a market research project ➡️ realize they are way over their heads ➡️ then scramble to find a professional market research firm to complete the project.
Over the years, the Drive Research team has completed thousands of market research projects and has responded to tens of thousands of leads, all of which are asking for help with their market research studies.
In this article, we'll talk about bad market research examples and some of the most common issues market research novices run into.
Example 1: Buying email lists and sending out surveys
If your email inbox is anything like mine, you likely receive dozens of monthly SPAM emails from companies trying to sell you vetted contacts.
They offer highly vetted decision-makers who fit various titles and roles.
These companies offer these lists of tens of thousands of email contacts for the low, low price of $99. The investment is so tiny that it’s a no-brainer for market research and your subsequent email survey project.
And many have gone down this path before.
Those list companies need to tell you that the open rates of these emails are tiny and getting the contact to click through to an offer deal or survey is minuscule.
All of that assumes the list of tens of thousands of connections is up to date.
With the average office worker receiving 121 emails every workday, it’s safe to say spam emails go straight to the trash without being opened.
Years ago, at another market research firm, we had a client approach us who wanted to leverage a purchased list of IT decision-makers for an email survey.
We went ahead and bought the list on their behalf for $250 and sent out 40,000 invites to IT decision-makers across the U.S.
Any guesses on the number of responses from that list?
Two.
Two whole responses.
The response rate was so low we went back to the email list vendor and asked them if they had done something wrong or did not send the email survey to the entire list.
They responded and said they did, but in good faith, they would send the email survey to a brand new list of $40,000 IT decision-makers in the U.S. at no additional charge.
How many responses did we get from that second list?
…Also two.
So we sent 80,000 emails and received 4 responses to our survey, a response rate of 0.005%.
Although the list purchase of $250 seemed like a no-brainer, with 2 responses from each list, we paid $125 per completed survey, which is 3X to 5X the cost you could get from a reputable market research panel provider.
I tell that story a lot because we’ve never returned to a list purchase since that day.
We have weekly conversations with potential clients at Drive Research to talk them out of that path.
In many unfortunate circumstances, they have already purchased the list, tried it independently, and are now scrambling to find 100 respondents for their management team presentation next week.
AKA, a classic example of bad market research.
Other disadvantages to purchased email lists
The other unspoken issue you run into when sending out emails to those who did not opt-in is running the risk of your domain getting black-listed.
Or at least generating a very high SPAM score.
This has major ramifications on future email surveys and general emails that are critical to your day-to-day operations.
Imagine trying to send an email that includes a $100,000 proposal to one of your clients where sign-off is due by the end of the day, and that email never makes it to their inbox.
All because an intern purchased a list of 80,000 dermatologists in the APAC region and sent an email survey last month through a Microsoft Outlook mail merge.
What to do instead
In market research, the only list that will provide a reasonable response rate to your email survey is your customer list.
Customer lists work better than purchased email lists because…
- The customer recognizes the sponsor who's sending the email
- The customer has some experience with them through a past purchase
Those two factors can generate a decent enough response rate for your market research project.
In fact, the average email open rate with customer lists is 21.5%, across all industries.
Other types of lists that are effective to a lesser degree include:
- Prospect lists from your CRM - A list of contacts who you may have reached out to before for sales purposes but they have not yet become a customer
- A list of former customers - Those who have since stopped doing business with you but will at least recognize your brand's name as part of the outreach
Anytime you send a cold email to a respondent who does not know your brand, you ask for trouble.
Furthermore, using a third-party market research firm or panel provider will ensure you get responses for your study.
Respondents on a market research panel often sign up to take surveys for incentives and rewards by offering up their contact information.
Thus, they have opted into and are expecting emails, which is a much different story than a cold email list you purchase and blast.
💡 The Key Takeaway: Purchasing email lists are an example of bad market research, as the idea of them sounds better than what they actually bring in. It’s a better idea to invest in a market research team that can proactively help you gather responses.
Recommended Reading: Stop Wasting Money on Purchased Email Customer Lists. Do This Instead.
Example 2: Not paying enough rewards for participant time
Another frequent error non-researchers make with market research studies is not paying enough incentives or rewards for the participants' time.
This issue is far too common in many requests from potential clients struggling to recruit participants for market research or get respondents to answer their survey.
It usually looks like an email survey to a list of engineers, and the brand would like to receive 400 responses.
They offer a single $50 Amazon gift card as part of a raffle to the engineers. However, an engineer's time is far too valuable for a 1 in 400 chance of winning a $50 gift card.
This example of bad market research is also widespread in the qualitative space when brands are looking to recruit physicians, business decision-makers, teachers, etc., to participate in a study of 90 minutes to 2 hours, yet can only offer $50 or $75 for their time.
The lack of response and engagement in these types of studies is evident.
When trying to recruit or have a highly specialized audience participate in your market research study, you have to pay them fairly for their time.
What to do instead
As part of your next study, consider creating a healthy budget for market research incentives or rewards for every respondent.
Trust us, you’ll thank yourself in the long run.
You’ll be generating more, higher-quality responses–creating a much more streamlined data collection window instead of stressing.
The recommended incentive for how much you should pay participants varies based on the audience and type of market research.
But, as a general rule of thumb, here are a few best practices to follow.
- B2C market research - On the consumer side, this often means starting with a $100+ reward for 30 to 60 minutes or more of their time.
- B2B market research With many specialized B2B audiences, this can often mean a starting point of $200 to $250+ for the same time.
Additionally, Drive Research has conducted market research with content creators and social media influencers for various brands.
This audience often demands upwards of $750 to $1,000 for a single hour of their time as part of an interview or focus group.
Raffles can often work with consumers, but the survey needs to be very short and engaging.
However, nothing in market research beats the pay-per-response model.
This could be something as simple as a $5 coffee gift card for every respondent on a 3-minute survey.
For instance, here is an example where G2 Crowd promotes their survey and the reward in a Facebook ad to earn responses.
There are a few ways to cap the number of responses so you stay within budget such as:
- Piece-mealing the list into small batches of invites to manage response rates
- Creating a quota to cap the number of responses in your survey by restricting access
- Explaining in the email that the first 100 responses will receive a reward
Unfortunately, the incentive for the study is far too often the biggest driver of a lack of response.
Pro Tip: Work with a professional market research firm so they can recommend how much you should pay your audience to thank them for their feedback and time as part of your next project.
💡 The Key Takeaway: Market research participants need to be compensated fairly for their time. Allocating part of your budget for this will ensure you receive high-quality responses.
Example 3: Long and tedious surveys
People are busier than ever. That's something people just say, though, but it is true.
Maybe people have lost their patience over the past few decades as instant gratification becomes more and more of a thing.
A funny study completed by Fifth Third shows just how impatient people are.
- 96 percent of Americans will knowingly consume extremely hot food or drink that burns their mouth
- More than half hang up the phone after being on hold one minute or less
- Nearly a third of respondents ages 18-24 wait less than one second before bypassing a slow walker
Bad market research can simply take too long, causing potential respondents to ignore it altogether.
Since 2016, Drive Research was set up to change that opinion.
We rarely run surveys longer than 10 to 15 minutes and consider 15 minutes to be borderline long.
However, we have seen many surveys over the years extending upwards of 30 to 45 minutes for consumers.
In those cases, you're setting your brand up for failure.
The failure either comes by not getting enough respondents to complete the survey.
Or the disappointment comes in respondents blitzing through the study and not paying attention (e.g., straight-lining responses or creating erroneous data just to finish faster).
Over the years, our firm has uncovered the science behind survey writing.
We find the “sweet spot” for survey length to be 5 to 7 minutes.
You can cover a lot of objectives in this time period, but the time to complete runs quickly for the respondent.
What to do instead
As a market research firm, we have always tried to balance client expectations with surveys and the respondent experience.
All in all, 5 to 7-minute surveys fit both well. Anything longer than that, and you have a “perfect” bad market research example.
If you have many objectives you want to cover in your market research study, consider sitting down to prioritize the core objectives and addressing those with your initial budget.
However, if you have additional funding, splitting your goals into 2 or 3 different surveys makes sense.
It may also create extra space for you to dive deeper into some of those secondary objectives.
Another part of survey writing is ensuring the respondent's experience is engaging and fresh.
For instance, changing question types by using…
- Various images
- Open ends
- Grids
- Scales
- And any or all of the above
It is also important to design your survey with a mobile-first mentality.
With over two-thirds of consumers taking surveys on smartphones or mobile devices, long grid questions with 15 rows or ranking questions with click-and-drag options are only sometimes the best fits.
When it comes to long grid questions, usually, each row is shown as a separate question on a mobile device. If you have a grid with 15 rows of ratings, that's 15 different screens the respondent will have to swipe through with responses.
Or, the columns for each row aren’t completely shown on the screen, similar to the visual below.
Image source: Qualtrics
If you are testing that survey on your desktop or laptop, it may all appear as a single grid or table, which seems more manageable.
Those questions can create a lot of frustration and increase your drop-off level. It is a common survey writing mistake to avoid at all costs.
With any good survey, it's essential to cover the objectives but also design a study that will be fun and engaging for the respondent to ensure you're collecting the highest quality data.
Looking for more mobile survey user experience best practices? Here are four tips to make your survey more mobile-friendly.
💡 The Key Takeaway: The moral of the story–keep surveys short! Think 5 to 7 minutes. This allows researchers to gather quality feedback without losing participants.
Example 4: Analysis and reporting without action
A good market research partner takes the story well beyond just the data.
Reporting back the data in tables and charts is not good enough these days.
Any firm can execute the market research process and spit out a banner file with numbers. But, the best market research companies tell you the what, so what, and now what of the insights.
Perspective and experience are critical.
Market research firms can often provide context from industry background or other similar clients (or both) to give your brand a bunch of added value with your analysis for your study.
The partner will tell you about the findings and what they recommend you do with those insights to improve operations, marketing, and strategy.
Better yet, many firms will break down a market research report so it’s digestible.
They’ll also talk to you about ways to maximize the ROI of your research by repurposing critical data points into…
- Sales presentations
- Website copy
- SEO keywords
- Blog content
- Whitepapers
- Lead generation gated content
- Webinars
For more thoughts here, read our post Want to Increase the ROI of Your Research? Publish it as Content.
💡 The Key Takeaway: A sign of bad market research is not giving clients the proper amount of background information about new data.
George Kuhn, Owner & President of Drive Research
Example 5: Collect survey responses via organic social media posts
When searching for responses for your survey, sharing it organically with your friends and family on Facebook, Instagram, or any other social media site is not suitable for several reasons.
The first reason is it will likely gain little traction.
If you're looking for chunks of responses, the number of people participating in your study from organic social shares is probably very low.
Many non-researchers have found this out the hard way before coming to a professional like Drive Research.
The second reason is responses from friends and family create a clear bias in your analysis.
A critical part of market research is to create a representative audience.
If you have only friends and family answering your survey, the results will likely be skewed. Particularly if you use quantitative research to measure or scientifically extrapolate the data to a larger audience.
So if you plan to rely on this source entirely for your survey, prepare to be disappointed.
What to do instead
While posting organic branded social media content to collect survey responses is an example of bad market research, paid social media ads are an excellent alternative.
Doing so allows you to share your survey with a large, more representative sample of people. It also offers various targeting tools to specifically promote the survey to those that match your ideal respondent.
This is a common recruiting tactic used by our market research company.
We go in-depth into this sampling source in our Ultimate Guide to Using Social Media to Find Research Participants.
Additionally, here is a webinar where our marketing manager and I chat more about the process and benefits of paid social media for market research recruiting.
💡 The Key Takeaway: Relying on organic social media shares alone will backfire and leave you with little data. Instead, work with panel vendors or invest in paid social media ads.
Example 6: Not quality-checking the data
You spent weeks collecting responses from hundreds of participants across the country. Now you can jump right into your analysis and report. Think again!
No matter what source you use to collect responses for your survey, you should always be quality-checking your data.
This informs your data-driven decision-making, which is how you end up using the provided feedback.
Quality-checking the data happens in three different phases:
- Identify and remove bad respondents
- Spot-check the data and remove cases as you go
- Run a final check on all of your responses to arrive at your final data set
Within the survey design, consider using some basic data quality-checking procedures. Here are a few examples.
Re-captcha
Things like a re-captcha to identify any bots trying to answer your survey. When it comes to surveys paying points or incentives, it gravitates towards automated bots racking up rewards.
Word associations
Some other more involved survey questions might be word associations. A question around which of these 5 words most closely relates to a shirt, with one of the options being pants.
Select all that apply
Another example is a select all that apply question, which asks the respondent to identify the three colors. You can decide to remove anyone who does not answer the list of colors provided.
Red herring
Red herring questions or categories later in the survey are also excellent ways to capture bad market research survey respondents.
A simple red herring question example could be including a brand name that does not exist and seeing which respondents select it.
Post-fieldwork data quality checks
Other data quality checks during fieldwork or after fieldwork include looking at the time to complete a survey.
Here are a few examples.
Outliers
If the survey took an average of 10 minutes to complete and you have a small percentage of survey takers who completed it in 2 or 3 minutes, you can remove them from your study and shave off the speeders.
Straight liners
Another option is to look at straightlining or those who select an answer rating “5” for a long grid question with 1-5 Likert scales in 10 consecutive rows.
If the respondent is genuinely engaged in the survey, there's likely to be some variance on a grid question with a series of Likert scales.
However, that should not be the only data quality check you're reviewing and deciding on. Straightliners should be coupled with the other data quality checks mentioned here.
Reviewing open ends
Another easy data quality check is simply reviewing the open-ended comments from a survey respondent.
You can easily spot some red flags in text-based responses that do not make sense from the question being asked.
With the emergence of Artificial Intelligence (AI), this is going to be much more challenging for research analysts to spot and identify.
That’s because, with AI, anyone can sound like a quick expert on any topic.
Again, this is why it is essential to use many data quality checks when running your survey and not rely on one factor.
💡 The Key Takeaway: A way to prevent bad market research from happening is by quality-checking the data. This ensures clients are only receiving the best responses.
Final Thoughts
Good market research is part art and part science.
The science is asking the right questions, eliminating bias, analyzing the results correctly, and creating a strategy from data-driven results.
The art is learning from past mistakes and understanding market research nuances that can only be defined through experience.
The examples of bad market research shared in this blog post often miss the mark on one or both of these elements.
By partnering with a third-party market research company, organizations can feel confident they are receiving 100% quality data with less time and internal resources wasted.
Contact Our Market Research Company
Working with a market research firm will help avoid all of these common pitfalls that those new to market research encounter.
A professional market research company will help fast-track your project to success ensuring you are designing the right instruments, using the right data collection techniques, and creating an impactful and actionable report to assist with your outcomes.
Drive Research is a market research company. Our team of pros has access to the right market research tools to design a high-quality project.
If you’d like to learn more about our market research services, get in touch with us today.
- Message us on our website
- Email us at [email protected]
- Call us at 888-725-DATA
- Text us at 315-303-2040
George Kuhn
George is the Owner & President of Drive Research. He has consulted for hundreds of regional, national, and global organizations over the past 15 years. He is a CX-certified VoC professional with a focus on innovation and new product management.
Learn more about George, here.
