Last active
December 22, 2024 15:21
-
-
Save mottalrd/7ddfd45d14bc7433dec2 to your computer and use it in GitHub Desktop.
Evan Miller source code for sample size from my blog post http://www.alfredo.motta.name/ab-testing-from-scratch/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
function num_subjects(alpha, power_level, p, delta) { | |
var t_alpha2 = ppnd(1.0-alpha/2); | |
var t_beta = ppnd(power_level); | |
var sd1 = Math.sqrt(2 * p * (1.0 - p)); | |
var sd2 = Math.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta)); | |
return (t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta); | |
} |
@sanjeevpe, I'm late but hope it helps. Below I present a customed Python function for estimating A/B testing sample size based on Evan Miller's sample size calculator. It includes the alternative to decide whether to used a two-tailed or one-tailed test or even relative or absolute delta.
Required modules
from scipy.stats import norm
import numpy as np
Evan Miller Sample Size Calculator
def evan_miller_sample_size(bcr, mde, sig_level = .05, power = .8, absolute_variation = True, two_tailed = True) -> int:
'''
This function retrieves the sample size required using Evan Miller's
methodology.
Args:
- bcr: the Baseline Conversion Rate.
- mde: the Minimum Detectable Effect (or practical significance).
- sig_level: the Alpha or Significance level (default 0.05).
- power: statistical power (default 0.8).
Returns:
- Required sample size per group.
'''
# Setting the variation type
effect_size = mde if absolute_variation else bcr * mde
p2 = bcr + effect_size
q1, q2 = 1 - bcr, 1 - p2
# Z-scores for significance level and power
z_alpha = norm.ppf(1 - sig_level / 2) if two_tailed else norm.ppf(1 - sig_level)
z_power = norm.ppf(power)
# Calculating the standard deviations
sd1 = np.sqrt(2 * bcr * q1)
sd2 = np.sqrt(bcr * q1 + p2 * q2)
# Computing the sample size per group
sample_size = pow(
(z_alpha * sd1 + z_power * sd2) / effect_size, 2
)
return round(sample_size)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
this is for relative delta. Is there a way to calculate sample size for absolute delta?
thank you