Bayes’ Theorem

Bayes’ Rule is an application of the LOTP with the Conditional Probability rule.

Bayes’ theorem describes the probability of an event based on prior knowledge of conditions that might be related to the event.

Let $B_1,\dots ,B_n$ be a partition of $S$ and $A$ be any event, then $P(B_i|A)=\frac{P(A|B_i)\cdot P(B_i)}{P(A)}=\frac{P(A|B_i)\cdot P(B_i)}{{\sum }_{R=1}^{n}P(A|B_R)\cdot P(B_R)}$

Alternate notation that is more useful for your ML stuff

$P(\theta \mid x)=\frac{P(x\mid \theta )\cdot P(\theta )}{P(x)}$

• $P(\theta \mid x)$ $\to$ Posterior Probability
• $P(\theta )$ $\to$ Prior Probability
• $P(x\mid \theta )$ $\to$ Likelihood

$x$ is your data and $\theta$ are your model parameters.

Proof of Bayes Theorem

How is bayes rule derived? Apply the basic rule of Conditional Probability, and leverage the fact that AND is commutative.

$P(B_i|A)=\frac{P(A\cap B_i)}{P(A)}=\frac{P(B_i\cap A)}{P(A)}=\frac{P(A_i|B)P(B)}{P(A)}$

Example of use of Bayes Theorem

Example

You have one fair coin and a biased coin that lands on heads with a probability of 3 4 . A coin is chosen at random and tossed three times. If we observe three heads in a row, what is the probability that the fair coin was chosen?

Solution: $B_1=$ The fair coin was chosen $B_2=$ The biased coin was chosen $A=$ 3 heads observed in 3 tosses

We want to find $P(B_1|A)$, so we can use Bayes’ Theorem, and calculate $P(A)=P(A|B_1)P(B_1)+P(A|B_2)P(B_2)$

Relation to more advanced control theory / ML

This is why we say (from Kalman Filter in Python) $Posterior=\frac{Likelihood\times Prior}{Normalization}$

• This is at the core of Bayes Filter update

You just continuously update the posterior, setting prior to old posterior every time.

See Kalman Filter.

Bayes theorem is super useful because it turns a hard problem into an easy problem.

Hard problems:

• P(Cancer = True | Test = Positive)
• P(Rain = True | Readings)

Stated like that the problems seem unsolvable.

Easy problems:

• P(Test = Positive | Cancer = True)
• P(Readings | Rain = True)

Bayes’ Theorem lets us solve the hard problem by solving the easy problem.

Bayes rule with Conditioning

From CS287. $P(x|y,z)=\frac{P(y|x,z)P(x|z)}{P(y|z)}$

Applied

https://x.com/cneuralnetwork/status/1870051432981819836

Related

• Conditional Probability
• Bayesian Inference
• Bayes Filter