Estimation

Once we can identify some sort of statistical model for our data, the goal is to estimate the parameters $\hat{\theta }(x_1,x_2,\dots ,x_n)$ of this statistical model. This process is called estimation.

Estimation = “What is our best guess for these unknown parameters?”

We use Estimators to do estimation.

Formally:

• The Problem: $X_1,X_2,\dots ,X_n$ is a sequence of i.i.d. r.v. with pmf $f(\theta )$, where $\theta$ is are the unknown true parameters
• Data: $\{x_1,x_2,\dots ,x_n\}$
• The Goal: to construct $\hat{\theta }(x_1,x_2,\dots ,x_n)$, which is an estimate of $\theta$ (true parameters)

Naive Approach

We can test various parameters $\hat{\theta }$. Suppose a coin is tossed with $\theta =P(Heads)$. We know that the coin is biased, where $\theta =\frac{1}{3}$ or $\theta =\frac{2}{3}$ (this is the $p$ parameter in the Binomial Distribution that we are trying to guess, $n=100$).

We run the experiment and we observe 60 heads. We can then “test” these parameters of the Binomial Distribution, where we find that:

$\theta =\frac{2}{3}⟹P(\text{observed data})=\left(\genfrac{}{}{0ex}{}{100}{60}\right){\left(\frac{2}{3}\right)}^{60}{\left(\frac{1}{3}\right)}^{40}=0.0307$

Therefore, you conclude that $\theta =\frac{2}{3}$ is more likely!

Now, instead of only these two parameters $\theta$ that you test, what if I asked you, out of all the possible parameters $\theta$ (which in this case, are ranging from $0.0$ to $1.0$, which one is the most likely?)

We can quantify the likelihood of a parameters with the Likelihood Function.

In this example, the likelihood function of a particular parameter given our observations would be $L(\theta )=\left(\genfrac{}{}{0ex}{}{100}{60}\right){\left(\theta \right)}^{60}{\left(1-\theta \right)}^{40}$ We want to find the parameters $\hat{\theta }$ that maximizes $L(\theta )$.

There exists multiple methods for Estimation:

• Minimum Message Length (MML)
• Maximum Likelihood Estimation (MLE)
• Method of Moments

In STAT206, we focus on MLE.