Bayes’ Theorem

Definition of Bayes’ Theorem

Bayes’ Theorem (or Bayes’ Law or Bayes' Rule) in probability theory and statistics states that the probability of event when a prior evidence or knowledge of conditions is given and that might be related to that event. For example, if cancer is related to age, then, using Bayes’ theorem, a person’s age can be used to more precisely or accurately assess the probability that they have cancer, in contrast to the assessment of the probability of cancer made without knowledge of the person's age.

Bayes’ theorem is a direct application of conditional Probabilities. Bayes’ theorem is useful, to determine posterior probabilities.
 

Formula of Bayes’ Theorem

Statement: Let E₁, E₂…E_n be the set of ‘n’ mutually exclusive and exhaustive events. Let A be any arbitrary event defined on the same sample space with P(A)≠0, then the probability of event E, when the event A has actually occurred is given by P(E_i /A)

Three types of probabilities occur in the above formula P (E_i), P(A/E_i), P(E_i/A)

• The probabilities P(E_i) i=1,2…..n are such that  P(E₁)+P(E₂)+….P(E_n)=1 are called prior probabilities since they are known before conducting experiment.

• The probabilities P (A/E_i) tell us, how likely the event A under consideration occurs, given each and every prior probability. They may be referred to as likelihood probabilities of the event A, given that event E_i has already occurred.

• The conditional probabilities P(E_i/A)  are called Posterior Probabilities, as they are obtained after conducting experiment.

Examples of Bayes’ Theorem

Q1. Cancer Age Problem:

Let’s assume that an individual’s probability of being affected by cancer, as per general condition, is 2%. This is known as the “Base Rate” or prior (i.e. before being informed about the particular case at hand) probability of being infected by cancer. Let C denote the event of "having cancer" Suppose that the probability of a person of being 70 years old is 0.2%. We write P (70) = 0.2%. We write P(70 | C) = 0.002. Finally, let us assume that cancer and age are related in the following way as: the probability for someone diagnosed with cancer with age 70 is 0.6%. This is written P(70 | C) = 0.006.

Considering this, we can calculate the probability of having cancer to a 70-year-old individual is P(C | 70), by applying Bayes' rule:

P(C | 70) = P (70 | C) P(C) / P (70) =0.006 * 0.02 / 0.002 = 6%

Possibly more instinctively, in a group of 100,000 people, 1,000 people will have cancer and 200 people will be 70 years old. Out of those 1000 people with cancer, only 5 people will be 70 years old. Thus, of the 200 people who are 70 years old, only 5 are expected to have cancer.

Even though an individual aged 70 years increases the risk of having cancer, that person’s probability of being affected by cancer is still fairly low, since the base rate of cancer (irrespective of age) is low. This concludes that both the importance of base rate, as well as that it is commonly neglected. Base rate neglect leads to serious misinterpretation of statistics; therefore, special care and attention should be taken to avoid such mistakes. Becoming familiar with Bayes’ theorem is one way to combat the natural tendency to neglect base rates.
 

Q2. A bag contains 7 red, 6 blue balls and another bag contains 6 red and 9 blue balls. A ball is drawn from the first bag and without knowing the color is put in the second bag. A ball is drawn from the second bag. Find the probability that ball drawn is blue in color.

Sol. Let event E₁: Red ball is drawn from the first bag and event E₂: Blue ball is drawn from the first bag.

P(E₁) = 7/13 P(E₂) = 6/15

(Note that E₁ and E₂ are exhaustive events and mutually exclusive)

Let event A:  Blue ball is drawn from the second bag.

P(A /E₁) = P(Blue ball is drawn from the second bag under the condition that red ball is transferred from the first bag to the second bag) = 9/15

Similarly, P(A/E₂) = P(Blue ball is drawn from the second bag under the condition that blue ball is transferred from the first bag to the second bag) = 10/15

Therefore, required probability= P (blue ball is drawn from the second bag)

P (A) = P (A∩E₁) + P (A∩E₂)

= P (E₁) P (A/E₁)+P(E₂)P(A/E₂)

 = 93/154
 

Q3. The chances of X, Y, Z becoming managers of a certain company are 4:2:3. If X, Y, Z become managers, the probabilities that the bonus scheme will bee introduced are 0.3, 0.5 and 0.8 respectively. If the bonus scheme has been introduced, what is the probability that X is appointed as the manager?

Let event E₁: Person X becomes manager

Event E₂: Person Y becomes manager

Event E₃: Person Z becomes manager

P (E₁) = 4/9 P(E₂) = 2/9  and P(E₃) = 3/9

(Note E₁, E₂ and E₃ are mutually exclusive and exhaustive events)

Let event A: Bonus introduced

P (A/E₁) = P(Bonus is introduced under the condition that person X becomes manager) = 0.3

P(A/E₂) = P(Bonus is introduced under the condition that person Y becomes manager) =0.5

P(A/E₃) = P(Bonus is introduced under the condition that person Z becomes manager) = 0.8

P(A) = P(A∩E₁) + P(A∩E₂) + P(A∩E₃)

= P(E₁)P(A/E₁) + P(E₂)P(A/E₂) + P(E₃)/P(A/E₃)

=23/45

Therefore, required probability = P(Person X becomes manager under the condition that bonus scheme is introduced)

P(E₁/A) = P(A∩E₁)/P(A) = P(E₁)P(A/E₁) / P(A)

= (2/15) / (23/45)

= 6/23
 

Q4. Balls in URN problem

There are two urns containing colored balls. The first urn contains 60 red balls and 60 blue balls. The second urn contains 40 red balls and 80 blue balls. One of these two urns is randomly chosen but both urns have probability of being chosen and then a ball is drawn at random from one of those two urns. If a red ball is drawn, what is the probability that it comes from the first urn?

Sol.  In probabilistic terms, what we know about this problem can be formalized as follows:

P(red | urn₁) = ½

P(red | urn₂)= 1/3

P(urn₁) = ½

P(urn₂) = ½

The unconditional probability of drawing a red ball can be derived using the law of total probability:

P(red) = P(red | urn1) P(urn1) + P(red|urn₂) P(urn₂)

= ½ . ½ + 1/3 .½

= ¼ + 1/6

= 6 + 4 / 24 = 5/12

By using Bayes' rule, we obtain

P(urn1 | red) = P(red | urn1)  P(urn1) / P(red)

= (½ ) . (½) / (5/12)

=( ¼ ) . (5/12)

= 5/24
 

Applications of Bayes’ Theorem

• Spam Filtering: This is one of the most widely and practically proven application of Bayesian inference. This refers to a program, when given a piece of Email can guess whether that E-mail is spam or not, based on the inference or data of previously received spam and non-spam E-mails. It generally uses certain words or phrases in the message to predict.

• In bio-chemistry, given the various blood sample tests, decision of occurrence of a particular disease is predicted, so it never be 100% true.

• In manufacturing , Selecting best products among two manufacturers.

• Another important approach to statistical inference is Bayesian inference. The probabilities may produce different interpretations or conclusions of probability when Bayesian inference is applied. This theorem describes how a subjective degree of belief should change logically when availability of prior related evidence is taken into consideration. It is fundamental to Bayesian statistics.

Some Other Examples of Bayes’ Theorem

Illustration:

A die is rolled and it is found that number turned up is an even number. Find the probability that it is 2.

Solution:

 All possible events when we roll a die are

A₁ : 1 appears P(A₁) = 1/6

A₂ : 2 appears P(A₂) = 1/6

A₃ : 3 appears P(A₃) = 1/6

A₄ : 4 appears P(A₄) = 1/6

A₅ : 5 appears P(A₅) = 1/6

A₆ : 6 appears P(A₆) = 1/6

An even number has turned up this is an information to us. You should revise the probability of priori events A_i in the light of information received. (It will be totally foolish if we don't revise the probabilities. Since the information of even numbers is available, it makes the probabilities of 1, 3, and 5 equal to zero).

 Let A: Even number has turned up.

We have to calculate the probability of A₂ when A is given. Since A₁, A₂, .........., A₆ are exhaustive and mutually exclusive we can apply Bayes’ formula

P(A₂/A) == (P(A)P(A/A₂))/(P(A₁ )P(A/A₁)+P(A₂ )P(A/A₂)+.……+P(A₆ )P(A/A₆))

where P(A/A₁) means probability of A when A₁ is given i.e. probability of coming of an event number when 1 has appeared. Obviously it is zero.

Hence P(A/A₁) = 0

Similarly, P(A/A₂) = 1, P(A/A₃) = 0, P(A/A₄) = 1, P(A/A₅) = 0, P(A/A₆) = 1

Therefore, P(A₂/A) = (1/6×1)/(1/6×0+1/6 ×1+1/6×0+1/6×1+1/6×0+1/6×1)=1/3

Hence this is the required probability.

Note: This problem is very simple and illustrated only to make the application of Bayes’ formula clear.
 

Illustration:

In a factory, machines A, B and C manufacture 15%, 25% and 60% of the total production of bolts respectively. Of the bolts manufactured by the machine A, B and C 4%, 2% and 3% are defective A bolt is drawn at random and is found to be defective. What is the probability that it was produced by B?

Solution:

In the formula P(A_k/A) = (P(A_k )P(A/A_k))/(∑ⁿ_(i=1) P(A_i ).P(A/A_i))

A_i means all the possibilities, which can happen w.r.t. to the given event A, while A_k means the particular event whose probability w.r.t. the event B we are required to find

Let us take A = the event of bolt being defective.

A₁ = the bolt is produced by B.

A₂ = the bolt is produced by A.

A₃ = the bolt is produced C.

Required probability = P(A₁/A)=(P(A₁ )P(A/A₁))/(∑³_(i=1) P(A_i ).P(A/A_i))

=>P(A₁/A) =(25/100×2/100)/((15/100×4/100)+(25/100×2/100)+(60/100×3/100))

= 50/(60+50+180)=5/29.

You may also refer the video for more examples on Bayes’ Theorem 

Illustration:

A bag contains 5 balls of unknown colors two balls are drawn at random and are found to be red. Find the probability that the bag contains exactly 4 red balls.

Solution:

First of all let's try to find out all the possibilities, which can occur w.r.t. to given events. Since two balls drawn are found to be red, so there are four possibilities. The bag contains only two red balls (say event A₂), bag contains 3 red balls (A₃), bag contains 4 red balls (A₄), or that all the balls are red balls (A₅).

Let B: two balls are drawn and found to be red. So, it is obvious that we have to find the probability of A₄ given B i.e. required probability = P(A₄/B)

 P(A₄/B)=(P(A₄).P(B/A₄))/(P(A₂).P(B/A₂)+P(A₃).P(B/A₃)+P(A₄).P(B/A₄)+P(A₅).P(B/A₅))

Now as all these four possible cases are equally likely i.e. we cannot say that bag is more likely to contains 5 red balls, than 3 red balls, so priori probabilities of all these events will be equal i.e.

P(A₂) = P(A₃) = P(A₄) = P(A₅)

Since A₂, A₃, A₄ and A₅ are exhaustive

.·.      P (A₂) + P (A₃) + P (A₄) + P (A₅) = 1

.·.     P (A₄) = ¼ (since A₂, A₃, A₄ and A₅ are equally likely also)

Now P (B/A₄) means that bag contains 4 red balls and we have to find the probability, that two balls drawn at random are red which is obviously ⁴C₂/⁵C₂ .

So P(A₄/B)=(1/4×(4C₂)/(5C₂))/(1/4×(2C₂)/(5C₂)+1/4×(3C₂)/(5C₂)+1/4×(4C₂)/(5C₂)+1/4×(5C₂)/(5C₂))

 = 12/39

Proof: This is so because

P(A_i/B)= P(A_i ∩ B)/P(B) = P(A_i ∩ B)/P((B ∩ A_i) U P(B ∩ A₂) U...(B ∩ A_n)) = P(A_i ∩ B)/P((B ∩ A₁) U P(B ∩ A₂) U...(B ∩ A_n))

= P (Ai).P (B/Ai)/∑ⁿ_i=1 P (A_i) P (B/A₁)

Note: In conditional probability the sample space is reduced to the set of samples/outcomes in the event which is given to have happened.

Illustration:

A rare genetic disease is discovered. Although only one in a million people carry it, you consider getting screened. You are told that the genetic test is extremely good; it is 100% sensitive (it is always correct if you have the disease) and 99.99% specific (it gives a false positive result only 0.01% of the time). Having recently learned Bayes' theorem, you decide not to take the test. Why?

Solution:

Bayes' Theorem states that for events X and Y:

P (X|Y) =P (Y|X)*P(X)/P(Y).

We want to know the probability of being healthy(X) given the positive test (PT) results(Y).

According to the Bayes' Theorem,

P (healthy|PT) =P (PT|healthy)*P (healthy)/P (PT).

From the problem we know that

P (healthy) =1-0.000001=0.999999

And getting a false positive

P (PT|healthy) = 0.0001.

The only unknown in the formula above is the probability of having a positive test P (PT). It can be calculated using the definition of marginal probability

P(Y)=P(Y|Z₁)*P(Z₁)+...+P(Y|Z_n)*P(Z_n),

where Z_i, i=1...n are all possible events. In our case there are only two possible events: "being healthy" and "being sick". Therefore

P (PT) =P (PT|healthy)*P (healthy) +P (PT|sick)*P (sick).

From the problem we know that

P (PT|sick) =1.0
(Test is always correct in presence of the disease) and

P (sick) =0.000001.

Substituting the numbers into the formula we get

P(PT)=0.0001*0.999999+1.0*0.000001=0.000101.

Finally,

P (healthy|PT)=0.0001*0.999999/0.000101=0.990098,
that is very close to 1.
So, the probability of still being healthy given that the results of the test turned positive is above 99%. That is a good reason for not taking the test.

Illustration:

An urn B₁ contains 2 white and 3 black balls and another urn B₂ contains 3 white and 4 black balls. One urn is selected at random and a ball is drawn from it. If the ball drawn is found black, find the probability that the urn chosen was B₁.

Solution:

Let E₁, E₂ denote the events of selecting urns B₁ and B₂ respectively.

Then P (E₁) = P (E₂) = 1/2

Let B denote the event that the ball chosen from the selected urn is black.

Then we have to find P (E₁/B).

Also, probability of selecting a black ball from the urn 1 i.e. P (B/E₁) is 3/5.

Probability of selecting a black ball from the urn 2 i.e. P (B/E₂) is 4/7.

Hence the required probability i.e. P(E₁/B) = P(E₁) P(B/E₁) / [P(E₁) P(B/E₁)+ P(E₂) P(B/E₂)

= ( ½. 3/5 )/ ½.3/5 + ½. 4/7

= 21/41. 

Illustration:

Each of three bags A, B, C contains white balls and black balls. A has a₁ white & b₁ black, B has a₂ white & b₂ black and C has a₃ white & b₃ black. A ball is drawn at random and is found to be white. Find the respective probability that it is from A, B & C.

Solution:

Here A₁, A₂, A₃ are the events that the bags picked are A, B, C respectively

E is the event that a white ball is drawn.

We are supposed to find P (A₁/E), P (A₂/E), and P (A₃/E).

P(A₁/E) = P(A₁UB)/P(E) = ((Prob. that bag A is chosen and white is drawn)/(Prob. that a bag is chosen at random and white is drawn))

= (P(A₁ ).P(E/A₁ ))/(P(A₁ ).P(E/A₁ )+P(A₂ ).P(E/A₂ )+P(A₃ ).P(E/A₃ ) )

= (1/3.a₁/(a₁+b₁ ))/(1/3.[a₂/(a₂+b₂ )+a₂/(a₂+b₂ )+a₃/(a₃+b₃ )] )=p₁/(p₁+p₂+p₃ )  

Similarly, P (A₂/E) = P₂/P₁+P₂+P₃ , P(A₃/E) = P₃/P₁+P₂+P₃

where p₁ = a₁/a₁+b₁ ,          p₂ = a₂/a₂+b₃ ,           p₃ = a₃/a₃+b₃ .

Illustration:

A bag contains 5 balls and of these it is equally likely that 0, 1, 2, 3, 4, 5 are white. A ball is drawn and is found to be white. What is the chance that is the only white ball?

Solution:

Here again, is a problem of conditional probability.

The condition B that is given is that one ball is drawn and is white.

Hence P(1w/B)=(P(1w/B).P(1w))/(P(0w)P(B/0w)+P(1w),p(B/1w)+..+P(5w).P(B/5w))

Where P(B/1w) - Probability that B occurs when exactly 1 w ball is there = ¹C₁/⁵C₁ and so on

P(1w) = P(2w) =………= P(5w) = P(0w) = 1/6 

=> the required probability = (1/6 (⁵ C₁)/(⁵ C₁))/(1/6 [0+(¹ C₁)/(⁵ C₁)+(² C₁)/( ⁵C₁ )+...( ⁵C₁)/( ⁵C₁ )] )=

 1/(0+1+2+...+5)=1/15.

You may also consult the Previous Papers to get an idea about the types of questions asked.

Illustration:

Find P (M/N), given P (N/M') = 0.7, P (M) = 0.2, and P (N/M) =0 .8.

Solution: P (M') = 1 - P (M) = 0.8.
By Bayes' Theorem,

P(M'/N)/P(M/N) = (P(N/M')P(M')/P(N))/(P(N/M)P(M)/P(N)) = 0.7*0.8/(0.8*0.2) = 7/2.
Hence 1 = P(M/N) + P(M'/N) = P(M/N)(1 + P(M'/N)/P(M/N)) = P(M/N)(1 + 7/2),
and so P(M/N) = 1/(1 + 7/2) = 2/9.

Illustration:

Find P (M'/N), given P (N/M') = 0.4, P (M) = 0.6, and P (N/M) =0.2.

Solution: P (M') = 1 - P (M) =0.4.
By Bayes' Theorem, P(M'/N)/P(M/N) = (P(N/M')P(M')/P(N))/(P(N/M)P(M)/P(N)) = .4*.4/(.2*.6) = 4/3.

Hence 1 = P(M/N) + P(M'/N) = P(M'/N)(P(M/N)/P(M'/N) + 1) = P(M'/N)(3/4 + 1),
and so P(M'/N) = 1/(3/4 + 1) = 4/7.

Watch this Video for more reference
 

Bayes’ Theorem is an important section of Probability. It is a simple and an interesting topic and fetches 2-3 questions in the exam. With a bit of hard work the topic can be mastered easily and it is important to excel in this topic to remain competitive in the IIT JEE.

More Readings

Bayes Theorem

To read more, Buy study materials of Probability comprising study notes, revision notes, video lectures, previous year solved questions etc. Also browse for more study materials on Mathematics here.