The first recalculation results in less perceived uncertainty, while the seconds results in higher perceived uncertainty. I used proper p-values and confidence intervals for % lift instead. P-values (statistical significance) and confidence intervals were calculated for absolute difference, while inferences were made for percentage change (% lift). One-sided calculations are obviously called for when making directional inferences.Ģ. Both significance and CIs are calculated with a two-sided instead of one-sided alternative hypothesis. This is due to two issues with the statistics as presented on the site:ġ. Unfortunately, one of the first things I had to do once I’ve manually extracted the number of users and conversions for each test was to recalculate the statistical significance (p-values) and confidence interval bounds. Statistical significance and confidence interval recalculations Note that this also contains the observed lift of tests that would not pass a 95% significance threshold. The median lift is 4.89%, meaning that 50% of the tests had an observed lift of less than 4.89%, the true lift could be even lower. We will get back to that in a few paragraphs. We can see that the results almost follow a normal distribution with mean lift 12.33% with the exception of a heavy right tail. The distribution of the observed effect sizes of the whole dataset in terms of percentage change looks like so: Power calculations are possible due to reported sample sizes. It also contains the observed percentage change. Statistical Recalculations & Pruningĭata on the GoodUI site comes with statistical significance calculations and confidence intervals, as well as their own assessment of how strong the result is, with possible values of “Insignificant”, “Possible”, “Significant” and “Strong”, going in both directions.

Clicking on each graph will open it zoomed to 100%, removing any blurring due to browser resizing. I will first discuss issues and necessary pruning done to the initial data set then share the detailed results of the resulting meta-analysis, followed by a brief discussion at the end. The GoodUI sample is likely not representative of all A/B tests done so any generalizations to the whole of A/B testing practice should be made with great caution. After pruning and statistical adjustments, the remaining 85 tests showed a mean and median lift of just under 4% relative change while the statistically significant ones had an average of 6.78% and median of 5.96%.

One of the major findings is that about 70% have low statistical power relative to expected treatment effects and economic logic, which hints at optional stopping issues. It is intended as an overview of average outcomes from a sample of A/B testing / conversion rate optimization activities, but more so as a guide to better statistical practices when designing and analyzing A/B tests. I set out to gather this data and I preformed a statistical meta-analysis of the outcomes of all 115 tests which is presented below in great detail. One source – has a slightly better approach in that for each of the tests* published on their website there is basic statistical information: the number of users and conversions exposed to each test variant, as well as what was tested ( example). Confidence intervals, p-values and other measurements of uncertainty will often be missing, and when present they are often poorly calculated and the statistical procedures are not shared and thus are hardly of any use. What can you learn from 115 publicly available A/B tests? Usually, not much, since in most cases you would be looking at case studies with very basic data about what was tested and the outcome of the A/B test. Oct 2022 update: A newer, much larger and likely less biased meta analysis of 1,001 tests is now available!

Statistical power formula