Contrastive Learning

Contrastive Loss

Resources

• https://lilianweng.github.io/posts/2021-05-31-contrastive/#contrastive-loss

Siamese Contrastive Loss

$L=(1-y)\cdot \frac{1}{2}{D}^2+y\cdot \frac{1}{2}{\left\{\max (0,m-D)\right\}}^2$

• $y=0$ if positive pair, $y=1$ if negative pair
• $D=\Vert f(x_1)-f(x_2){\Vert }_2$ is the euclidean distance
• $m$ is the margin, how far apart the they must be

So if they’re a positive pair, we minimize the left term. If they’re a negative term, they must be at least $m$ apart.

Used for example in Siamese Networks and CLIP.

CLIP Contrastive Loss (InfoNCE)

Given a batch of $N$ image–text pairs $(\{(x_i^{\text{img}},x_i^{\text{text}}){\}}_{i=1}^N)$:

• Image encoder: $f_{\theta }(x^{\text{img}})$
• Text encoder: $g_{\varphi }(x^{\text{text}})$
• Normalize to unit vectors:

$$z_i^{\text{img}}=\frac{f_{\theta }(x_i^{\text{img}})}{\Vert f_{\theta }(x_i^{\text{img}})\Vert },z_i^{\text{text}}=\frac{g_{\varphi }(x_i^{\text{text}})}{\Vert g_{\varphi }(x_i^{\text{text}})\Vert }$$

• Similarity score:

$$s_{ij}=z_i^{\text{img}}\cdot z_j^{\text{text}}$$

Image → Text loss:

$$L_{\text{image}\to \text{text}}=-\frac{1}{N}\sum _{i=1}^N\log \frac{\exp (s_{ii}/\tau )}{{\sum }_{j=1}^N\exp (s_{ij}/\tau )}$$

• sum over all texts

Text → Image loss:

$$L_{\text{text}\to \text{image}}=-\frac{1}{N}\sum _{i=1}^N\log \frac{\exp (s_{ii}/\tau )}{{\sum }_{j=1}^N\exp (s_{ji}/\tau )}$$

• $sum$ over all images

Total CLIP loss (symmetric):

$$L=\frac{1}{2}(L_{\text{image}\to \text{text}}+L_{\text{text}\to \text{image}})$$

• (\tau): learnable temperature parameter.
• Numerator: similarity of true image–text pair.
• Denominator: similarity across all candidates in batch.
• Effect: each image must “classify” its paired text, and vice versa.

SigLIP

SigLIP removes the softmax and instead uses a sigmoid + binary cross-entropy formulation.