Cross-Entropy Loss

We use this loss for Classification problems. LLMs use this loss to be trained on predicting next tokens.

Cross-entropy measures the difference between two probability distributions.

Resources

• https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss

The cross-entropy between a “true” distribution $p$ and an estimated distribution $q$ is defined as: $H(p,q)=-\sum _{x}p(x)\log q(x)$

Where does this log come from?

This comes from Shannon Entropy.

He proved that under certain very reasonable assumptions, log is the only possible choice.

In Classification

We can go from −x∑​p(x)logq(x)→−logq(yn​) $-\sum _{x}p(x)\log q(x)\to -\log q(y_n)$

• p(x) vanishes into a One-Hot distribution

From pytorch $\ell (x,y)=L=\{{\ell }_1,\dots ,{\ell }_N{\}}^{\top }$

• $N$ spans the minibatch dimension (i.e. there are $N$ samples in a batch). If this is already confusing to you, see a more basic example in L1 Loss.

There are 2 different formulations depending on how the classes are predicted.

If you use indices as the target: where

$${\ell }_n=-w_{y_n}\cdot \log \left(\frac{\exp (x_{n,y_n})}{{\sum }_{c=1}^C\exp (x_{n,c})}\right)\cdot 1\{y_n\ne \text{ignore\_index}\}$$

This was really confusing to me.

• $w_{y_n}$ is an optional weight matrix, useful when you have an unbalanced training set
• $x_{n,c}$ is the value of the logit at index $c$
• $x_{n,y_n}$ is the value of the logit as the correct index

$\text{softmax}(x_n)=\frac{\exp (x_{n,y_n})}{{\sum }_{c=1}^C\exp (x_{n,c})}$

${\ell }_n=-w_{y_n}\log \left(\text{softmax}(x_{n,y_n})\right)$

• The Negative Log Likelihood of the correct output

Is this a similar idea of using the negative log to the Negative Log Likelihood??

Also see Softmax Classifier

If you use

Related