A/B STATISTICAL SIGNIFICANCE CALCULATOR
Calculate statistical significance
Determine whether the difference between two results is significantly significant.
HOW TO USE THE STATISTICAL SIGNIFICANCE CALCULATOR
Enter the results of an A/B test
Start by selecting a confidence level in your results, then enter the visitors and conversions for two cases of a recent split test, and we’ll calculate the statistical significance between those two cases for you.
Statistical significance calculator FAQs
Everything you need to know about calculating statistical significance for A/B tests.
What is statistical significance?
Statistical significance is a concept used in hypothesis testing to determine whether an observed difference between groups is likely to be a real effect or if it could have occurred by chance.
In other words, it helps researchers assess whether the results of an experiment or study are meaningful and not simply due to random variation.
In A/B testing, statistical significance is crucial for determining whether there is a meaningful difference between two versions of a product, webpage, marketing strategy, or any other variable being tested. So, you can see why finding statistical significance also works well as an A/B testing calculator.
A/B testing involves comparing the performance of two or more variants (A and B) to see which one performs better in terms of a specified metric (e.g., conversion rate, click-through rate, revenue).
Perform A/B tests in surveys with our Pro plan: Plans & pricing.
How to calculate statistical significance
- Collect Data: Gather data on the number of visitors and conversions for both variants (A and B) of your A/B test.
- Calculate Conversion Rates: Compute the conversion rates for each variant by dividing the number of conversions by the number of visitors.
- Find the Difference in Conversion Rates: Determine the difference in conversion rates between the variants (e.g., Conversion Rate for B – Conversion Rate for A).
- Calculate Standard Error: Compute the standard error, which measures the variability of the sample mean, using the pooled conversion rates of both variants.
- Calculate Z-score: Calculate the Z-score, which represents the number of standard deviations the observed difference is from the expected difference under the null hypothesis.
- Determine Critical Z-values: Based on the desired confidence level (e.g., 90%, 95%, 99%), find the critical Z-value(s) from a standard normal distribution table or use pre-calculated values.
- Find the p-value: Use the Z-score to calculate the p-value, which represents the probability of observing the difference (or more extreme) if there were no real difference between the variants. If you don’t have access to the p-value, you can compare your z-score directly to the critical z-score of your confidence level.
- Interpret Results: Compare the calculated p-value to the chosen significance level (usually 0.05 or 0.01). If the p-value is less than or equal to the significance level, the results are considered statistically significant, indicating a meaningful difference between the variants.
Need to calculate sample size? You’re in luck, check out our sample size calculator.
What is p-value?
In the simplest terms, p-value tells us the probability of obtaining the observed results by random chance.
If the p-value is small (usually less than a pre-determined significance level, commonly 0.05), it suggests there is a statistical significance between the two values.
On the other hand, if the p-value is large, it suggests that the observed results are reasonably likely that the difference could have been achieved by random chance.
What is confidence level?
The confidence level is typically expressed as a percentage and represents the probability that the true parameter value lies within a certain interval.
For example, a 95% confidence level means that if we were to repeat our experiment or sampling process many times, we would expect the true parameter to fall within the calculated interval about 95% of the time.
A higher confidence level implies a wider interval for estimating the parameter, but it also requires stronger evidence to claim statistical significance.
You can use our critical z-score table below to determine the score associated with each confidence level.
| Confidence level | Significance level | Critical z-score |
|---|---|---|
| 90% | 0.1 | 1.65 |
| 95% | 0.05 | 1.96 |
| 99% | 0.01 | 2.58 |
Get more survey responses with Shout
Build beautiful, responsive, surveys that help you collect more feedback.
Global targeting
Find the perfect respondents for your surveys using a huge variety of targeting criteria with Shout Audience.
Omnidirectional feedback
Maximize your response rates by sharing your survey via email invitation, website embed, QR code, URL link, and more.
Beautiful designs
Customize every element of your survey’s design to ensure it stands out to respondents and keep them engaged.
Personalize everything
Build surveys with personal information about your respondents and increase responses rates by up to 48%.
Send reminders
Import respondents to our CRM to track responses and send gentle reminders to those yet to respond to your survey.
Incentivise responses
Offer incentives to those who complete your survey and send them to a promotional page using our Exit Page Redirect.