May 19, 2020

FFR040 - How To Tell When A Split Test Result Is Statistically Significant

FFR040 - How To Tell When A Split Test Result Is Statistically Significant

One of the most common questions I get is how do you know when to declare a winner in a split test... and also how much traffic do you need to send to it.    One thing to keep in mind is the more traffic you can send to a split test the...


  • One of the most common questions I get is how do you know when to declare a winner in a split test... and also how much traffic do you need to send to it. 

 

  • One thing to keep in mind is the more traffic you can send to a split test the better... as if one page has only 12 views and the other page has 8 views you can’t call that a proper split test.

 

  • But really the amount of hits a page has got is irrelevant, what really matters is whether the result is statistically significant or not. If it’s not then the experiment must continue until it is. 

 

  • As a general rule I won’t even look at split results unless there is a sample size of at least 100 at the very minimum. And even more so if the conversion rates are close. 

 

  • You see, just because the control or the variation has got off to flying start doesn’t mean it will continue that way. It’s quite possible that the other version will pick up steam and reserve the result. 

 

  • And terminating a split test too early because it seems obvious who has won is known as a false positive.

 

  • So with said... the question is how can you get an accurate answer without needing to ask someone who has a PhD in Statistics.