What is Statistical Significance?

Why should you care about statistical significance?

It’s no longer enough to post an ad or publish a website and hope it drives conversions; welcome to the age of conversion rate optimisation (CRO).

Case in point: during the 2008 Obama campaign, the Director of Analytics, Dan Siroker and the campaign team ran multivariate tests to test 6 different images and 4 Call to Actions (CTAs) for a single landing page – that’s 24 variations in total – hoping to generate more sign-ups. The test was responsible for an additional 2,880,000 email addresses, translating into an additional $60 million in donations. You can read more about that particular test on Siroker’s blog, but stay with us while we dive into statistical significance.

One of the important factors in conversion rate optimisation is statistical significance – yet sadly, online resources for statistical significance are generally rife with examples that aren’t at all aligned with business goals, like the size of red apples VS the size of green apples. Such trite examples can make it difficult to comprehend the value and importance of running tests for your business or side hustle and measuring statistical significance.

For example: currently your website’s contact us page has a CTA of “Contact me” and you’d like to see if a CTA of “Get in touch” results in more visitors submitting an enquiry. You set up an A/B test and you want to know how your test is performing and if your results are conclusive. Simply calculating the conversion rate of each test isn’t enough; you also need to be reasonably confident that any differences in conversion rate between your two variations are due to your change and not just luck.

Once you calculate the confidence of the change between the conversion rate, you’ll know if you should complete your test and apply your winning variation to 100% of the visitors to the page (or even create a new A/B test, using your winning variant as your new control).

This could be a critical decision for you to make, leading to hundreds more conversions than you would have otherwise seen. Even if you’re not a numbers person, that’s data to get excited about – one change directly attributed to more dollars in your back pocket!

That’s how much of a difference testing can make to your bottom-line results – and statistical significance is the essential concept to understand the results of your split testing efforts.

What is statistical significance and what does it mean?

If you’re wondering how statistical significance relates to CRO, it boils down to three words: the null hypothesis.

Similar to the legal phrase, ‘innocent until proven guilty’, the null hypothesis is assumed to be true unless evidence indicates otherwise. The damning evidence that might slam your null hypothesis behind bars is known as the alternative hypothesis.

Statistical significance is the measure of confidence used to determine whether your null hypothesis is guilty of being false, and how likely it is that the results of your test are due to chance. In other words, statistical significance is the mathematical equivalent of a lawyer’s ‘beyond reasonable doubt’.

Like the phrase ‘beyond reasonable doubt’, whether a test is statistically significant is also a point of contention – some people say that a confidence level of 95% is required; others hold out for at least 99%. Ultimately, this is up to you: but the higher the score, the less likely it is that your result is due to chance.

To take a step back here, let’s talk about confidence levels and how these relate to probability values (or p-values). If you’re not a mathematician, it can be slightly confusing to understand how confidence levels relate to p-values. In short, the probability value is the inverse of your confidence level.

So, if you have a statistically significant result that indicates a 99% confidence level, the probability value (AKA the probability that your result is due to random chance) is 1%.

Now you know the what and why of statistical significance, let’s talk about how to calculate it easily. *No apples included.

How to easily measure statistical significance

There are many articles out on the wide vast web that will give you complicated formulas or supposed Excel hacks (still complicated) to determine statistical significance, but it doesn’t need to be that complicated.

All you need to determine statistical significance is at least two variations and a population size and the conversions for each of the variations.

Depending on the test you’re running, these numbers could look a little different. For example

Once you've got these metrics, you can use our free A/B test statistical significance calculator to determine the statistical significance of your test.