In this lesson, we shall look at the law of total probability. In probability theory, the law (or formula) of total probability is a fundamental rule relating marginal probabilities to conditional probabilities. It expresses the total probability of an outcome which can be realized via several distinct events, hence the name.
You will be able to:
- Understand and explain the concept of event space and partitioning
- State the law of total probabilities based on a partitioned event space
- Understand and able to perform partitioning based on known and unknown probabilities to solve a problem
- Understand and describe conditional independence
the Law of Total Probabiliyy can be used to calculate P(B). The law requires that you have a set of disjoint events Ai that collectively "cover" the event B. Then, instead of calculating P(B) directly, you add up the intersection of B with each of the events Ai. Let's see this graphically below:
Let
The probability of a random event
\begin{align} P(B) &= P(B \cap A_1) + P(B \cap A_2) + \dots + P(B \cap A_n) \ &= P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + \dots + P(B|A_n)P(A_n) \end{align}
Here we use the first theorem mentioned in the previous lesson to find the combined probabilities.
Let's try to understand this using a simple example.
In a certain country there are three provinces; B1, B2, and B3 (i.e., the country is partitioned into three disjoint sets B1, B2, and B3). We are interested in the total forest area in the country. Suppose that we know that the forest area in B1, B2, and B3 are 100 km2, 50 km2, and 150 km2, respectively. What is the total forest area in the country?
100km2 + 50km2 + 150km2 = 300 km2
We can simply add forest areas in each province (known as a partition) to obtain the forest area in the whole country. This is the idea behind the law of total probability, in which the area of forest is replaced by probability of an event A. In particular, if you want to find P(A), you can look at a partition of S (our sample space), and add the amount of probability of A that falls in each partition.
We can say that for any two events A and B:
and using the definition of conditional probability, P(A∩B)=P(A|B)P(B), we can write
We can state a more general version of this formula which applies to a general partition of the sample space S.
If B1,B2,B3,⋯ is a partition of the sample space S, then for any event A we have
Using a Venn diagram, we can pictorially see the idea behind the law of total probability. In the figure below, we have
- A1=A∩B1
- A2=A∩B2
- A3=A∩B3
$P(A)=P(A1)+P(A2)+P(A3)$
Here is a typical scenario in which we use the law of total probability. We are interested in finding the probability of an event A, but we don't know how to find P(A) directly. Instead, we know the conditional probability of A given some events Bi, where the Bi's form a partition of the sample space. Thus, we will be able to find P(A) using the law of total probability
Above, you saw that we solved a more complicated problem by looking at this event as a combination of smaller event. You'll see this happen a lot in probability theory.
Decomposing sets into pieces is often done via a Partition. Recall this definition of sets:
A partition of a set
$X$ is a collection of subsets$Xi \in X$ so that every element$x \in X$ occurs in exactly one of the$Xi$ .
- The natural numbers
$\mathbb{N}$ can be partitioned into even and odd numbers. - The set of animal species in the world can be partitioned into subsets where a subset reflects a continent and each species is positioned in a subset depending on which continent they originated from.
In statistics choosing the right partitioning is key as bad choices of partitions may results in many, even more difficult to solve sub-problems.
The total probability law allows us to break up probability calculations into distinct parts. It’s used to find the probability of an event, A, when you don’t know enough about A’s probabilities to calculate it directly. Instead, you take a related event, B, and use that to calculate the probability for A.
The probability for a can be written as sums of event B. The total probability rule is:
$P(A) = P(A∩B) + P(A∩B^c)$
An alternate version of the total probability rule (found with the multiplication rule) can be used when the necessary probabilities are known :
$P(A ∩ B) = P(A | B) * P(B) + P(A ∩ B^c) = P(A | B^c)P(B^c)$
ONe really needs to be careful when dealing with conditional probabilities and conditioning. Let's look at a few examples to see this idea in action.
In a certain county, 60% of registered voters are Republicans, 30% are Democrats and 10% are Independents.
When those voters were asked about increasing military spending
- 40% of Republicans opposed it
- 65% of the Democrats opposed it
- 55% of the Independents opposed it.
What is the probability that a randomly selected voter in this county opposes increased military spending?
Let:
- Ω = {registered voters in the county}
- R = {registered republicans}, P(R) = 0.6
- D = {registered democrats}, P(D) = 0.3
- I = {registered independents}, P(I) = 0.1
- B = {registered voters opposing increased military spending}
We also know:
- P(B|R) = 0.4
- P(B|D) = 0.65
- P(B|I) = 0.55
By the total probability theorem:
= (0.4 · 0.6) + (0.65 · 0.3) + (0.55 · 0.1) = 0.49.
Let's consider a 2-card hand drawn from a standard playing deck. What is the probability of drawing 2 aces, given that we know one of the cards is an ace?
\begin{align} P(\text{both are aces | one is ace}) &= \frac{P(\text{both are aces})}{P(\text{one is ace})} \ &= \frac{P(\text{both are aces})}{1 - P(\text{neither is ace})} \ &= \frac{\binom{4}{2}/\binom{52}{2}}{1 - \binom{48}{2}/\binom{52}{2}} \ &= \frac{1}{33} \end{align}
But now think about this: What is the probability of drawing 2 aces, knowing that one of the cards is the ace of spades?
\begin{align} P(\text{both are aces | ace of spades}) &= P(\text{other card is also an ace}) \ &= \frac{3}{51} \ &= \frac{1}{17} \end{align}
Notice how the fact that we know we have the ace of spades nearly doubles the probability of having 2 aces.
Suppose there is a test for a disease, and this test is touted as being "95% accurate". The disease in question afflicts 1% of the population. Now say that there is a patient who tests positive for this disease under this test.
First we define the events in question:
Let
Let
Since that phrase "95% accurate" is ambiguous, we need to clarify that.
\begin{align} P(T|D) = P(T^c|D^c) = 0.95 \end{align}
In other words, conditioning on whether or not the patient has the disease, we will assume that the test is 95% accurate.
What exactly are we trying to find?
What the patient really wants to know is not
\begin{align} P(D|T) &= \frac{P(T|D)P(D)}{P(T)} ~~~~ & &\text{... Bayes Rule} \ &= \frac{P(T|D)P(D)}{P(T|D)P(D) + P(T|D^c)P(D^c)} ~~~~ & & \text{... by the Law of Total Probability} \ &= \frac{(0.95)(0.01)}{(0.95)(0.01) + (0.05)(0.99)} ~~~~ & & \text{... the rarity of the disease competes with the rarity of true negatives}\ &\approx 0.16 \end{align}
- Mistaking
$P(A|B)$ for$P(B|A)$ .
This is also known as the Prosecutor's Fallacy, where instead of asking about the probability of guilt (or innocence) given all the evidence, we make the mistake of concerning ourselves with the probability of the evidence given guilt.
- Confusing prior
$P(A)$ with posterior$P(A|B)$ .
Observing that event
- Confusing independence with conditional independence.
This is more subtle than the other two. Let's look at this in a bit more detail
Events
\begin{align} P(A \cap B | C) = P(A|C)P(B|C) \end{align}
i.e. conditioning on event
Consider playing a series of 5 games against a chess opponent of unknown strength. Winning all five games gives would give you a good idea that you are better player. So winning each successive game actually is providing us with information about the strength of our opponent. If you have prior knowledge about the strength of your opponent, you condition on the strength of our opponent i.e. Winning one game would not provide us with any additional information on the probability of winning the next. Having no prior knowledge of your opponent, and winning a string a games will give you information about the probability of winning the next game.
The games are conditionally independent given the strength of our opponent, but not independent unconditionally.
For example Let
So
You are strongly advised to visit following links to get an indpeth understanding with examples and proofs for formulas highlighted in this lesson.
The law of total probability - concept and proof - Excellent YouTube video by Phil Chan.
Conditional (Partitioned) Probability — A Primer - Deep dive into partitions (A Must read)
Law of Total Probability - More examples for a deeper understanding around partitioning
In this lesson, we further built on the ideas of conditional probability covered in the previous lesson to explain the law of total probability using partitioning of the sample space. We saw how we can partition probabilities with respect to some other event, when the direct probabilities are not known. Next we shall look at the "Bayes Theorem" which uses all the ideas covered so far and adds more goodness to the probability calculation.