Skip to content

Instantly share code, notes, and snippets.

@mottalrd
Last active December 22, 2024 15:21
Show Gist options
  • Save mottalrd/7ddfd45d14bc7433dec2 to your computer and use it in GitHub Desktop.
Save mottalrd/7ddfd45d14bc7433dec2 to your computer and use it in GitHub Desktop.
Evan Miller source code for sample size from my blog post http://www.alfredo.motta.name/ab-testing-from-scratch/
function num_subjects(alpha, power_level, p, delta) {
var t_alpha2 = ppnd(1.0-alpha/2);
var t_beta = ppnd(power_level);
var sd1 = Math.sqrt(2 * p * (1.0 - p));
var sd2 = Math.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta));
return (t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta);
}
@mysticaltech
Copy link

This is missing from the part before the declaration of t_alpha2, see https://www.evanmiller.org/ab-testing/sample-size-fixed.js

if (p > 0.5) {
        p = 1.0 - p;
}

@sanjeevpe
Copy link

this is for relative delta. Is there a way to calculate sample size for absolute delta?
thank you

@domingosdeeulariadumba
Copy link

domingosdeeulariadumba commented Dec 22, 2024

@sanjeevpe, I'm late but hope it helps. Below I present a customed Python function for estimating A/B testing sample size based on Evan Miller's sample size calculator. It includes the alternative to decide whether to used a two-tailed or one-tailed test or even relative or absolute delta.

Required modules

from scipy.stats import norm
import numpy as np

Evan Miller Sample Size Calculator

def evan_miller_sample_size(bcr, mde, sig_level = .05, power = .8, absolute_variation = True, two_tailed = True) -> int:
'''
This function retrieves the sample size required using Evan Miller's
methodology.

Args:
- bcr: the Baseline Conversion Rate.
- mde: the Minimum Detectable Effect (or practical significance).
- sig_level: the Alpha or Significance level (default 0.05).
- power: statistical power (default 0.8).

Returns:
- Required sample size per group.
'''
    
# Setting the variation type    
effect_size = mde if absolute_variation else bcr * mde
p2 = bcr + effect_size
q1, q2 = 1 - bcr, 1 - p2 

# Z-scores for significance level and power
z_alpha = norm.ppf(1 - sig_level / 2) if two_tailed else norm.ppf(1 - sig_level)        
z_power = norm.ppf(power)

# Calculating the standard deviations
sd1 = np.sqrt(2 * bcr * q1)
sd2 = np.sqrt(bcr * q1 + p2 * q2)

# Computing the sample size per group
sample_size = pow(
    (z_alpha * sd1 + z_power * sd2) / effect_size, 2
)    
return round(sample_size)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment