If you have ever encountered market research data, chances are you have heard the phrase, “statistically significant.” It sounds great, but what does it actually mean?
Statistical significance in market research refers to the likelihood that observed differences or relationships between variables are not due to random chance but are meaningful and reliable.
By applying significance testing in market research, researchers can gain insights into the effectiveness of different marketing approaches or product variations, helping them make informed decisions to optimize their marketing efforts and improve business outcomes.
Read on to get an overview of what statistical significance is, how it works, examples, and what forms it comes in.
What is Statistical Significance in Market Research?
Significance testing is a statistical method that has been widely used within the market research world. A significant result refers to the probability that the difference between two or more data points is unlikely the result of chance.
If we feel confident that the difference is not just a random coincidence, then we say it is meaningful. A meaningful difference can be used by clients to help make more informed decisions about their business.
You can conduct significance testing on a set of proportions (i.e., the percentage of respondents who select an option) or a set of means (i.e., the average number of something among respondents).
Statistical significance styles
There are also two main styles of significance testing in market research: pairwise testing and contrast testing.
Pairwise testing
Pairwise testing compares a data point individually with every other data point in the group or segment.
The advantage here is you can identify the presence of significant survey results between any two subgroups (i.e., Millennials and Baby Boomers).
This is the type of testing our market research firm uses by default for clients in cross-tabulations and banner files.
Contrast testing
Contrast testing compares a data point to an aggregate accounting for all the other data points in the group or segment.
This approach may be useful if you are more concerned with how significant one subgroup’s responses are versus all the rest combined (i.e., Millennials versus all other age groups).
What is an Example of Significance Testing in Market Research?
One example of significance testing in market research is conducting an A/B test to compare the performance of two different marketing strategies or variations of a product.
In this scenario, the goal is to determine whether the observed differences in outcomes (such as sales, conversion rates, or customer engagement) between the two groups are statistically significant or simply due to random chance.
In another example, it is common to identify many statistically significant results when analyzing respondent data by a subgroup (i.e., age, household income) or wave of research (i.e., brand tracking studies).
For example, you may learn that within the context of your study, male respondents are significantly more likely to participate in an activity than female respondents.
What Are The Types of Significance Testing?
There are two types of significance tests you will regularly see in market research: one-tailed tests and two-tailed tests. Significance testing also takes various shapes depending on what you are testing. This may include Z-tests and Chi-squared tests.
Let’s explore each of these a bit further.
One-tailed tests
As the name implies, one-tailed tests focus on statistical significance in a single direction. This means a one-tailed test informs us whether a data point is significantly larger OR significantly smaller than another data point, but not both.
Two-tailed tests
In most cases, two-tailed tests are a more valid approach for significance testing for market research data.
Two-tailed tests inform us if a data point is significantly different from another, and whether the data point is significantly higher or lower than its counterpart.
These are the default test types for our firm’s cross-tabulations because they factor in both sides of the probability distribution curve.
Z-tests
Z-tests are the most common type of test for identifying significant relationships between two proportions or percentages for most sample sizes.
A Fisher’s Exact test is used instead for cases with small sample sizes. T-tests are used to determine significant differences between two or more means.
Chi-squared tests
Chi-squared tests are another type of significance test used to look for a broader relationship between your row variable and column variables in a cross-tabulation of proportions. This won’t test for differences between individual data points like a z-test, but it does offer a quick way to identify what variables might be connected.
How to Determine Statistical Significance?
Probability is at the root of significance testing.
Because it is impossible to say with 100% certainty that two data points have a significant difference, we rely on probabilities to inform us how likely it is that the difference is meaningful.
We also must assume the data has a normal distribution (bell curve) as a prerequisite for significance testing.
Here's a step-by-step breakdown of how significance testing can be applied in market research:
1. Hypothesis formulation
The researcher starts by stating a null hypothesis (H0) and an alternative hypothesis (H1).
For example, the null hypothesis could be that there is no significant difference in sales between two different marketing strategies, while the alternative hypothesis would state that there is a significant difference.
2. Sample selection
A market research company, such as Drive Research, divides the target population into two groups:
- Group A, which represents one marketing strategy or product variation
- Group B, which represents the other.
The groups should be similar in characteristics and size to minimize confounding factors.
3. Data collection
Relevant metrics or performance indicators are tracked for each group over a specified period. For instance, sales revenue, website traffic, or conversion rates might be recorded.
4. Statistical analysis
Various statistical tests can be employed to assess the significance of the observed differences between the groups.
As we discussed above, commonly used tests include t-tests, chi-square tests, or analysis of variance (ANOVA), depending on the nature of the data and research question.
5. Determining statistical significance
The statistical testing provides a p-value, which indicates the probability of observing the observed differences (or more extreme) between the groups under the assumption of the null hypothesis.
A commonly used threshold for significance is p < 0.05. A general rule of thumb is if the p-value is less than the significance level, the difference between data points is significant.
6. Conclusion and implications
Based on the results, the researcher can conclude whether there is a statistically significant difference in the performance of the marketing strategies or product variations.
When the p-value meets this requirement, we can reject what is called the null hypothesis. The null hypothesis for most significance testing in market research is that the two data points being tested are equal.
Discovering the data points are unlikely equal is how we ultimately define statistical significance.
Statistical Significance FAQs
Why is .05 statistically significant?
In statistical hypothesis testing, the significance level of 0.05 (or 5%) is commonly used as a threshold for determining statistical significance because it represents a reasonable balance between Type I and Type II errors, where rejecting the null hypothesis (H0) when it is true (Type I error) and failing to reject H0 when it is false (Type II error) are both considered undesirable but unavoidable risks in statistical inference.
What are P-values?
P-values are key outputs in statistical significance tests. If you were wondering, the “p” in “p-value” actually stands for probability. P-values are typically calculated by computers these days for more accuracy.
- The closer a p-value is to zero, the more likely it indicates a significant difference between two data points.
- The closer a p-value is to 1, the more likely any observed difference is just the result of chance.
P-values are tested based on a chosen confidence interval to determine whether the difference is significant. Confidence intervals can range from as high as 99.9% to as low as 70%.
What is the risk of using a lower confidence interval?
As you reduce the confidence interval, you increase the chance of identifying a significant difference when there is nothing really there. This is a situation known as Type 1 error. Be wary of market research results that use low confidence intervals, as they may be artificially boosting the statistical significance of their data to fit a narrative.
Here are four other ways to test the credibility of market research data.
Why doesn’t every research study use the highest possible confidence interval?
Another issue referred to as Type II error makes this an ill-advised decision. Type II error occurs when you overlook a significant result because the confidence interval is set too high, missing a potentially important finding.
This all tells us that no confidence interval is perfect. However, the market research standard used in most studies is a confidence interval of 95%. It is accepted as a good sweet spot for identifying reliably significant results.
Final Thoughts
Statistical testing plays a crucial role in market research by providing a robust framework to evaluate and draw meaningful insights from data.
By setting a predetermined significance level, such as 0.05, researchers can objectively assess whether the observed results are statistically significant or merely due to random chance.
It enables researchers to assess the significance of observed differences, relationships, or experimental outcomes. This ultimately guides organizations toward data-driven strategies and successful outcomes in the dynamic world of business.
Contact Our Market Research Company
Drive Research is a national market research company located in Syracuse, NY. Every set of data tables our firm delivers for clients features comprehensive statistical significance testing to help you quickly find out what is important within your data.
Interested in learning more about our market research services? Reach out through any of the four ways below.
- Message us on our website
- Email us at [email protected]
- Call us at 888-725-DATA
- Text us at 315-303-2040
Tim Gell
As a Senior Research Analyst, Tim is involved in every stage of a market research project for our clients. He first developed an interest in market research while studying at Binghamton University based on its marriage of business, statistics, and psychology.
Learn more about Tim, here.
