Likelihood

Negative Log Likelihood

First, understand likelihood and understand that likelihood is just Joint Probability of the data given model parameters $\theta$, but viewed as a function of $\theta$, i.e. $L(\theta )$.

One way to do Maximum Likelihood Estimation is to minimize the negative log likelihood.

That is ${\max }_{\theta }{\mathbb{E}}_{x\sim p_{data}}[\log p_{\theta }(x)]={\min }_{\theta }-{\mathbb{E}}_{x\sim p_{data}}[\log p_{\theta }(x)]$

Log Likelihood

Log likelihood makes the math a lot easier: log(a*b*c) = log(a) + log(b) + log(c)

We do this because a*b*c might be an extremely small number, so we perform addition instead.

Likelihood:

$$\mathcal{L}(\theta )=\prod _{i=1}^{n}p(x_i\mid \theta )=\mathbb{E}?$$

• ❌ Not expressible as expectation

Log-likelihood (Average): $\log \mathcal{L}(\theta )=\frac{1}{n}{\sum }_{i=1}^n\log p(x_i\mid \theta )={\mathbb{E}}_{x\sim p_{data}}[\log p(x\mid \theta )]$

Negative log-likelihood (Average): $-\log \mathcal{L}(\theta )=-\frac{1}{n}{\sum }_{i=1}^n\log p(x_i\mid \theta )=-{\mathbb{E}}_{x\sim p_{data}}[\log p(x\mid \theta )]$

Can we not just minimize negative likelihood instead?

Like do we HAVE to use the log? We would not be able to do Cross-Entropy loss because its not consistent with their formula (if you use log it is). But we can somehow derive another formula for the optimization?

• In theory yes, but in practice makes your life 10x harder at the moment?
• In theory yes, but in practice makes your life 10x harder at the moment.

The issue is that you’d be optimization

$$-\mathcal{L}(\theta )=-\prod _{i=1}^{n}p(x_i\mid \theta )$$

Multiplying the probabilities is going to result in numerical stability issues.

Connection with Cross-Entropy

You will notice that the formula is actually the same as the Cross-Entropy definition! ${\min }_{\theta }-{\mathbb{E}}_{x\sim p_{data}}[\log p_{\theta }(x)]=\mathbb{H}(p_{data},p_{\theta })$ Note that $p_{\theta }(x)=p(x\mid \theta )$ (I use both notations).

In PyTorch, we have both CrossEntropyLoss and NLLLoss. They’re actually the same thing, but NLLLoss expects the input to already be softmaxed (probability computed) and logged.

LogSoftmax layer + NLLLoss == CrossEntropyLoss

Optimization Landscape

Was trying to understand how using log likelihoods help with the optimization landscape as well.

🔧 Setup: Binary Classification Toy Example

Let’s imagine a simple model outputs a probability $p_{\theta }(x)\in (0,1)$ of a binary label being 1. Suppose the true label is $y=1$. We’ll compare two loss functions: ${\mathcal{L}}_{\text{raw}}=-p_{\theta }(x)$ and ${\mathcal{L}}_{\text{log}}=-\log p_{\theta }(x)$.

Now we’ll plot both loss functions as a function of the predicted probability $p_{\theta }(x)\in (0,1)$.

You can see that the gradients of -log(p) are much smoother.

In Robotics

See Behavior Cloning. It’s essentially Cross-Entropy Loss.