ab_testing_math.md

A/B Testing Statistics

Find the average click through rate:

We're assuming the opposite, so we're assuming there's one average click through rate for both pages:

$$\widehat{\mu} = \frac{k_a + k_b}{n_a + n_b} = \frac{10 + 13}{100 + 100} = \frac{23}{200} = 0.115 $$

Now we also have to find the variance of the click through rate:

You can look up the mean and variance of a distribution on Wikipedia. This is the distribution of a single visitor and whether this visitor signs up (1) or not (0).

$$\widehat{\sigma}^2 = 0.115 * (1 - 0.115) = 0.101775$$

Find the standard error (standard deviation of the mean)

• The variance of the mean is the variance of the individual variable divided by the number of samples.
• The variance of the difference of two independent variables is the SUM of the variances of each of those variables.

$$SE^2 = SE_a^2 + SE_b^2 = \frac{\sigma_a^2}{n_a} + \frac{\sigma_b^2}{n_b}$$

• Combine terms

$$SE^2 = \frac{\widehat{\sigma}^2}{n_a} + \frac{\widehat{\sigma}^2}{n_b}$$

$$SE^2 = \widehat{\sigma}^2 \left( \frac{1}{n_a} + \frac{1}{n_b} \right) $$

• Take the sqrt to go from variance to standard deviation

$$SE = \widehat{\sigma} \sqrt{\frac{1}{n_a} + \frac{1}{n_b}} $$

$$SE = 0.045116515822922316$$

Find the "z" score, the standard normal parameter:

$$z = \frac{\mu_a - \mu_b}{SE}$$

$$z = \frac{0.13 - 0.10}{0.045116515822922316} $$

$$ z = 0.664944964228774 $$

Do a TWO sided test:

$$ final = P[Z>z] + P[Z<-z]$$

$$ final = 1 - CDF(z) + CDF(-z) = 2 * \left(1 - CDF(z)\right) $$

$$ final = 2 * \left(1 - CDF(z)\right) $$

Go back to Javascript:

final = 2 * (1 - jStat.normal.cdf(0.664944964228774, 0, 1))

And We're Done:

$$ final = 0.5060856949 $$