Maximum Likelihood Estimation

Likelihood Function

The likelihood function (often simply called the likelihood) is the Joint Probability of the observed data viewed as a function of the parameters of the chosen statistical model.

Key idea for MLE: $\text{Likelihood}=P(x\mid \theta )$

• $x$ is your data
• $\theta$ are your model parameters

Likelihood Function (Definition)

If $Y_i\sim f(y_i,\theta ),i=1,2,\dots ,n$, where $Y_i$ are i.i.d. RVs with observations $\{y_1,y_2,\dots y_n\}$, then $L(\theta ;y_1,y_2,\dots y_n)=P(Y_1=y_1,Y_2=y_2,\dots ,Y_n=y_n)={\prod }_{i=1}^{n}f(y_i,\theta )$

• $f$ is the Probability Density Function
• $\theta$ are the parameters we are trying to estimate, depends on the distribution. Ex: for Gaussian Distribution, $\theta =(\mu ,{\sigma }^2)$

Probability vs. Likelihood

• Probability is assigning the probability of a data value given distribution, i.e. $P(data|distribution)$
• Likelihood is the probability of a distribution given data values, i.e. $L(distribution|data)$

https://www.youtube.com/watch?v=pYxNSUDSFH4&ab_channel=StatQuestwithJoshStarmer

Negative Log Likelihood

First heard from Andrej Karpathy.

Log likelihood: $L(\theta )=\frac{1}{m}\sum \log p_{\theta }(x^i)$

We use the log because the probabilities can be very small, so we work with Log Function.

We negative it so the value can be positive on domain 0 to 1.

One super neat trick from the Log Rules is that instead of multiplying everything, we can just add all the logs, i.e. log(a*b*c) = log(a) + log(b) + log(c)

We do this because a*b*c might be an extremely small number, so we perform addition instead.

How likelihood is used

I’m still trying to wrap my head around this. But essentially, you use a series of likelihood updates.

Your prior is your belief distribution. Then, you have new observations. NO, you are getting confused.

The belief distribution refers to a probability distribution over possible outcomes or states, typically representing subjective probabilities based on a person’s knowledge or judgment.

Based on your belief distribution, you update your prior.

• I’m still confused

Our goal is to find the posterior.

Example:

• Based what you observe with the measurements, update the position (state) of the dog
• position is the prior and posterior
• measurement is used to update the prior. But where’s the likelihood in all of this?

This chapter is it: https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/02-Discrete-Bayes.ipynb

• Likel

Related

• Bayes Rule
• Relative Likelihood Function
• Normal Distribution
• Probability