Bayes' Theorem By/For Idiots

gistfile1.txt

# Bayes' Theorem

Let's pretend that you wish to find the probability that two events, A and B, occur.  
If A and B are independent events, then probability that A and B both occur is 

    P(A)P(B).

However, A and B might be related events. If they are not independent, the probability that A and B both occur is

    P(A)P(B|A)

That is, the probability that A and B both occur is the probability that A occurs, times the probability that B occurs given that you know that A has occured. The probability of A can also be conditioned on the occurence B:

    P(B)P(A|B) = P(A)P(B|A)

Bayes' Theorem:

    P(A|B) = ( P(A)P(B|A) ) / P(B)
    P(B|A) = ( P(B)P(A|B) ) / P(A)

Where:

* P(A|B) is the *posterior*
* P(A) is the *prior*
* P(B|A) is the *likelihood*
* P(B) is the *normalizing constant*. The normalizing constant ensures that the probability is valued between 0 and 1. If you are only interested in the likelihood of a result given a sequence of evidence, you do not need to normalize the probability.

## The Cookie Jar Example

Let's pretend that you have two jars of cookies; `Jar 1` contains `10 chocolate chip cookies` and `30 sugar cookies`. `Jar 2` contains `20 chocolate chip cookies` and `20 sugar cookies`. Your friend takes a `sugar cookie` from one of the jars. If you wish to find which jar your friend mostly likely took the cookie from, you might say that jar 1 is more likely, since it contains a greater proportion of sugar cookies than jar 2. Bayes' Theorem lets you calculate how much more likely it is that your friend took the cookie from jar 1.  

Now let's define the problem more formally:

* Hypothesis 1 (H1): Your friend took the cookie from jar 1.
* Hypothesis 2 (h2): Your friend took the cookie from jar 2.
* Prior probabilities: P(H1) = P(H2) = 0.5. Your friend randomly selected the jar, so both are equally likely.
* Likelihoods: 
 * P(E|H1) = 0.75: The probability of taking a sugar cookie from jar 1 is 0.75, since the jar contains 3/4 sugar cookies and 1/4 chocolate chip cookies.
 * P(E|H2) = 0.5: The probability of taking a sugar cookie from jar 2 is 0.5, since the jar contains half sugar cookies and half chocolate chip cookies.
* P(E) - 0.625: The probability of the evidence. The probability of taking a sugar cookie under any circumstances is 0.625, since of the 80 cookies contained in both bowls, 50 are sugar cookies.
* P(H1|E) = ( P(H1)P(E|H1) ) / P(E) = ( 0.5 * 0.75 ) / 0.625 = 0.6.

The probability that your friend took the cookie from jar 1 is 0.6.

In this example, you could directly observe P(E). In other cases, you might not be able to observe P(E) directly. Recall that P(E) is the probability that E happens; if you cannot observe P(E) directly, you can instead sum the probabilities of all scenarios in which E occurs. In this example, there are two scenarios in which E occurs: hypothesis 1 and hypothesis 2.

    P(E) = P(H1)P(E|H1) + P(H2)P(E|H2) = 0.5*0.75 + 0.5*0.5 = 0.625

More generally, P(E) is the sum of the each hypothesis's prior times its likelihood.

## An example from Wikipedia

Your friend tells you that she had a pleasant conversation with someone, and that this person had long hair. You wish to guess the gender of the long-haired person your friend conversed with, but don't wish to ask your friend for any additional information, because you have a peculiar personality.

Initially, you can assume that the probability that the mysterious conversation participant is a woman is 50% (P(W) = 0.5), since we are pretending that the population is composed of 50% men and 50% women. However, your friend also mentioned that the participant had long hair. If you know that 75% of women have long hair (P(L|W) = 0.75), and that 15% of men have long hair (P(L|M) = 0.15), you can use Bayes' Theorem to calculate the probability that your friend conversed with a woman.

P(W|L) = ( P(L|W)P(W) ) / P(L) = ( P(L|W)P(W) ) / ( P(L|W)P(W) + P(L|M)P(M) ) = (0.75 * 0.5) / ((0.75 * 0.5) + (0.15 * 0.5)) =  0.375 / .45 = 0.8333