Bayes Theorem

http://community.topcoder.com/tc?module=Static&d1=tutorials&d2=mmProblems2

Bayes Theorem

By timmac

In the case of LumberjackExam, where the probabilities and errors refer to yes/no values, we can take a more precise mathematical approach to dealing with the inaccuracy by using some rules that are formally demonstrated by Bayes' Theorem.  The theorem can be stated in a variety of different ways, but here's the one I find most useful:

Suppose a set of conditions, A1, A2,..., An each independently suggest that a condition B exists with probabilities p1, p2,..., pn, then the overall probability that condition B exists is given by:

So how does this apply? Let's consider a real world example.  Suppose that, of all people with long hair, 80% are computer programmers.  Suppose also that of all the people who wear glasses, 70% are computer programmers.  Now, suppose someone walks by who has long hair, and wears glasses.  What are the chances he is a computer programmer?

At first glance, someone might confuse this with another type of problem and arrive at an answer of 94%. I'm referring to problems like, "There is a 70% chance of rain today, and an 80% chance of rain tomorrow.  What are the chances that it rains today or tomorrow?"

The key difference here, in the example of the rain problem, is the word "or." It doesn't matter if it rains today, tomorrow, or both. When dealing with conditional probabilities that fall under Bayesian rules, that same "or" does not apply; that is, the two propositions are no longer independent events. The visually impaired, long-haired gentleman in question is either a programmer or he is not -- thus, either both conditionals are true, or both are false.

If both are true, then 70% * 80% = 56%.  If both are false, and he is not a programmer, then we evaluate 30% * 20% = 6%.  Now, notice that 56% + 6% does not add up to a full 100%.  So, we divide both by 62% to scale the value appropriately.  Thus, there is approximately a 90.3% chance this gentleman is a programmer.

There is one more important and useful detail of this basic equation.  Though I will leave the formal algebraic proof as an exercise to the reader, compare what happens if we have three conditional probabilities (as opposed to just two), and one of them is the calculated overall probability based upon two or more conditionals.  Perhaps not surprisingly, the result is the same.  What this means for us is that, using Bayes' theorem, we can accurately keep track of the probability of something being true by repeatedly applying the theorem every time we get a new piece of information, without having to keep track of the individual conditional probabilities that we have previously collected.

In the example above, we already know the gentleman has a 90.3% chance that he is a programmer.  If we were then to be told that whenever someone is carrying a laptop (and let's assume our friend is), that they have a 60% chance of being a programmer, we can then update our probability:

p = (0.903 * 0.6) / ((0.903 * 0.6) + (.097 * 0.4)) = 93.3%.

This is particularly useful in the lumberjack problem.  Every time we make an observation, we calculate a conditional probability for each tree (which was included in that viewing) being infected.  If any of those trees have already been viewed, then we can use the theorem to update our conditional probability for any of those trees, and can continue to do so with each new observation.  And, in theory, we can continue doing so repeatedly, each time ending with a more accurate feel for if the tree is actually infected.  In the case of this particular problem, we can make some observations about the effectiveness of looking at certain long distances, and may wish to only consider, say, the first five trees in front of us.

During the Intel competition series, the MessageReceiver problem had a similar conditional probability at work, in that each letter could either be transmitted correctly or incorrectly.  Knowing this, it should come as no surprise that the winning solution relied heavily upon Bayes' theorem as well.