Maximum Likelihood Estimation

Likelihood Function

The likelihood function (often simply called the likelihood) is the Joint Probability of the observed data viewed as a function of the parameters of the chosen statistical model.

“Probability of what you see given your model”

$L(\theta )=P(x_1,x_2,\dots x_n,\mid \theta )={\prod }_{i=1}^{n}P(x_i\mid \theta )={\prod }_{i=1}^{n}f(x_i\mid \theta )$

• where $f$ is the Probability Density Function

So likelihood is just Joint Probability?

When we use the term likelihood, we’re not treating $\theta$ as fixed.

Key idea for MLE: $\text{Likelihood}=P(x\mid \theta )$

• $x$ is your data
• $\theta$ are your model parameters

The other way around is the Posterior.

“It can be shown1 that minimizing the KL Divergence is equivalent to minimizing the Negative Log Likelihood, which is what we usually do when training a classifier, for example.

• YES I think I finally get that

Likelihood Function (Definition)

If $X_i\sim f(x_i,\theta ),i=1,2,\dots ,n$, where $X_i$ are i.i.d. RVs with observations $\{x_1,x_2,\dots x_n\}$, then $L(\theta ;x_1,x_2,\dots x_n)=P(X_1=x_1,X_2=x_2,\dots ,X_n=x_n)={\prod }_{i=1}^{n}f(x_i,\theta )$

• $f$ is the Probability Density Function
• $\theta$ are the parameters we are trying to estimate, depends on the distribution. Ex: for Gaussian Distribution, $\theta =(\mu ,{\sigma }^2)$

Probability vs. Likelihood

• Probability is assigning the probability of a data value given distribution, i.e. $P(data|distribution)$
• Likelihood is the probability of a distribution given data values measures how well a distribution explains the data, i.e. $L(distribution|data)=P(data|distribution)\ne P(distribution|data)$,

Probability treats data as variable, and likelihood treats parameters as variable.

https://www.youtube.com/watch?v=pYxNSUDSFH4&ab_channel=StatQuestwithJoshStarmer

Related

• Bayes Rule
• Relative Likelihood Function
• Normal Distribution
• Probability