Likelihood Estimation

Maximum Likelihood Estimation (MLE)

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data.

https://www.youtube.com/watch?v=XepXtl9YKwc&ab_channel=StatQuestwithJoshStarmer

title: Definition (The Maximum Likelihood Estimate (MLE))  
$\widehat{θ}$ is the MLE if $\widehat{θ}$ maximizes $L(θ; y_1, y_2, \dots y_n)$ where $L$ is the [[Likelihood Function]].

General Template for Deriving MLE

We always use the log likelihood since it makes it much easier to derive. See Logarithm Rules, but basically you have that $\ln (abc)=\ln a+\ln b+\ln c$

You then take derivative, and set that to 0, since you want to maximize it.

We model binomial with bernouilli, so binomial is a special case. We just do $X\sim Bin(n,p)$

But for the other distributions, we state $X_1,\dots ,X_n\sim Exp(\theta )$ for example. So I am a little confused on this?

We define

• $L(\theta )$ as the likelihood
• $l(\theta )$ as the log likelihood

Binomial MLE

Suppose $Y\sim Bin(n,\theta )$, with $y$ observed successes. Then what is $\hat{\theta }(y)$?

Likelihood $L(y;\theta )=P(Y=y)=\left(\genfrac{}{}{0ex}{}{n}{y}\right){\theta }^y(1-\theta {)}^{n-y}$ Let’s derive $\hat{\theta }(y)$ by using the log likelihood. Maximizing log likelihood is the same as maximizing likelihood. $l(\theta )=\ln L(\theta )$ $\hat{\theta }$ maximizes $L(\theta )⟺\hat{\theta }$ maximizes $l(\theta )$ $l(\theta )=\ln k+y\ln \theta +(n-y)\ln (1-\theta )$ $\frac{dl(\theta )}{d\theta }=0⟹\frac{y}{\hat{\theta }}-\frac{n-y}{1-\hat{\theta }}=0$ $⟹\hat{\theta }=\frac{y}{n}$

• For the Binomial Distribution, the parameter is simply the sample proportion of success $p=\frac{\#\text{ observed successes}}{\#\text{ total trials}}$, which intuitively should make sense.

Poisson MLE

Let $Y_1,Y_2,\dots ,Y_n\sim Poi(\theta )$ with observations $\{y_1,\dots ,y_n\}$. What is the MLE of $\theta$? $\hat{\theta }=\frac{1}{n}\sum y_i=\overline{y}$

• Remember for Poisson Distribution the parameter is $\lambda$, and $E(y)=\lambda$, so the parameter is simply the mean.

I got practice deriving this, and it seems that

Exponential MLE

$\hat{\lambda }=\frac{1}{\overline{y}}$

• Remember that if $X\sim Exp(\lambda )$, then $E(X)=\frac{1}{\lambda }=\mu$, so the parameter $\lambda =\frac{1}{\mu }$

Normal MLE

Suppose $Y_1,\dots ,Y_n\sim N(\mu ,{\sigma }^2)$ with observations/data of $\{y_1,\dots y_n\}$

What is the MLE of $\mu$ and ${\sigma }^2$? $\hat{\mu }=\overline{y}$ ${\hat{\sigma }}^2=\frac{1}{n}\sum (y_i-\overline{y}{)}^2$ $s^2=\frac{1}{n-1}\sum (y_i-\overline{y}{)}^2$ Am I supposed to use the $n-1$ or the $n$?? I supposed $n-1$ because that we are estimating the variance of a sample, or a population? We are estimating the variance of a population.

Derivation for Normal MLE

We use the definition of Likelihood:

&={\frac {1}{\sigma^n (2\pi)^\frac{n}{2} }}e^{-{\frac {1}{2\sigma^2}}\sum \left({y_i-\mu }\right)^{2}} \\ &={\frac {1}{\sigma^n} \cdot \frac{1}{(2\pi)^\frac{n}{2} }} \cdot e^{-{\frac {1}{2\sigma^2}}\sum \left({y_i-\mu }\right)^{2}} \\ l(u, \sigma^2) &=- n \log \sigma - \frac{n}{2} \log 2\pi - \frac{1}{2\sigma^2} \sum(y_i - \mu)^2 \end{align}$$ This likelihood is maximized when the derivative is 0 (similar ideas in [[notes/Least Squares|Least Squares]]). ### Properties of the [[notes/Maximum Likelihood Estimation|MLE]] For discrete -> $L(\theta)$ the probability of observing $\theta$ For continuous - > recall [[notes/Probability Mass Function|p.m.f.]] = gives probability directly -> [[notes/Probability Density Function|p.d.f.]] -> $f(y_i)$ is not a probability 1. Consistency - As $n \rightarrow \infty$, $\widehat{\theta} \rightarrow \theta$ (our estimate converges to the true value) 2. Efficiency - We want a minimum variance when finding $\widehat{\theta}_i$ 3. [[notes/Invariance|Invariance]] - If $\widehat{\theta}$ is the MLE of $\theta$, then $g(\widehat{\theta})$ is the MLE of $g(\theta)$ Other Notes - We assume that the class of the distribution has been properly identified - We assume that we have [[notes/independent and Identically Distributed|i.i.d]] datasets ### Related - [[notes/Likelihood Function|Likelihood Function]] - [[notes/Relative Likelihood Function|Relative Likelihood Function]]