Representation Learning

Contrastive Learning

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other, while dissimilar ones are far apart.

Resources

• https://lilianweng.github.io/posts/2021-05-31-contrastive/
• https://www.youtube.com/watch?v=7l6fttRJzeU&t=318s&ab_channel=ArtificialIntelligence
• I first First seen from here: https://github.com/wjf5203/VNext

Ohh, I actually found about about this, I spoke with the people at Cohere AI at HackWestern.

We have the following categorization

1. Inter-sample classification (most dominant)
   • Given both similar (“positive”) and dissimilar (“negative”) candidates, to identify which ones are similar to the anchor data point is a classification task
2. Feature Clustering
   • Find similar data samples by clustering them with learned features
3. Multiview coding
   • Apply the InfoNCE objective to two or more different views of input data

The CLIP model enables Zero-Shot classification.

There are creative ways to construct a set of data point candidates:

1. The original input and its distorted version
2. Data that captures the same target from different views

Some loss:

• Contrastive Loss (works with labelled dataset)
• Triplet Loss
• N-Pair Loss (generalizes triplet loss)
• Lifted Structured Loss
• Noise Contrastive Estimation (NCE)
• InfoNCE - see contrastive loss, used by CLIP
• Soft-Nearest Neighbors Loss

Resources

• https://encord.com/blog/guide-to-contrastive-learning

Some contrastive models

• SimCLR
• MoCo
• CLIP

Framework (CS231n 2025 Lec 12)

Given an anchor $x$, a positive $x^{+}$ (semantically similar — typically another augmentation of $x$), and negatives $x_j^{-}$ (other samples), learn an encoder $f(\cdot )$ and a score function $s(\cdot ,\cdot )$ such that:

$s(f(x),f(x^{+}))\gg s(f(x),f(x^{-}))$

InfoNCE loss (van den Oord 2018) frames this as an $N$-way softmax classification — pick the positive out of 1 positive + $N-1$ negatives:

$L=-{\mathbb{E}}_X\left[\log \frac{\exp (s(f(x),f(x^{+})))}{\exp (s(f(x),f(x^{+})))+{\sum }_{j=1}^{N-1}\exp (s(f(x),f(x_j^{-})))}\right]$

Minimizing $L$ maximizes a lower bound on mutual information between anchor and positive:

$MI[f(x),f(x^{+})]-\log (N)\ge -L$

So more negatives → tighter MI bound → better representation (intuition behind why SimCLR/MoCo push hard for large $N$).

Instance vs sequence contrastive

Level	Positives from	Examples
Instance	two augmentations of the same image	[[research-papers/A Simple Framework for Contrastive Learning of Visual Representations
Sequence	future timesteps of the same sequence given past context	Contrastive Predictive Coding (CPC)

SimCLR (Chen 2020)

• Given a minibatch of $N$ images, draw two augmentation functions $t,t^{\prime }\sim \mathcal{T}$ → produce $2N$ augmented views.
• Encode each with $f(\cdot )$ → $h_i$, then project with $g(\cdot )$ → $z_i\in {\mathbb{R}}^D$. At inference throw away $g(\cdot )$, keep only $h$.
• Score = cosine similarity $s_{i,j}=z_i^T{z}_j/(\Vert z_i\Vert \Vert z_j\Vert )$ — builds a $2N\times 2N$ affinity matrix; positive pair for view $i$ is its partner at $2N\times 2N$ offset.
• InfoNCE over the $2N$ rows of the affinity matrix.
• Non-linear projection head $g$ is crucial — representation space $h$ stays richer when the invariance pressure is applied in a separated $z$ space.
• Large batch crucial — batch 8192 beats batch 256 by ~5 points on ImageNet linear eval. Requires TPU pods for memory.
• Top-1 ImageNet linear eval: SimCLR (4×) 76.5%, matching supervised ResNet-50.

CPC (van den Oord 2018) — sequence-level

1. Encode each timestep: $z_t=g_{\text{enc}}(x_t)$.
2. Summarize context: $c_t=g_{\text{ar}}(z_{\le t})$ using an autoregressive model (original paper uses GRU-RNN).
3. InfoNCE between context $c_t$ and future code $z_{t+k}$ with time-dependent score: $s_k(z_{t+k},c_t)=z_{t+k}^T{W}_k{c}_t$ $W_k$ is a trainable matrix, one per future offset $k$. Negatives are codes from other sequences / random positions.

Applied to audio (LibriSpeech — 64.6% phone, 97.4% speaker linear probe vs supervised 74.6% / 98.5%), and to images by raster-scanning 64×64 patches with 50% overlap (top-down context predicts bottom rows).

MoCo (He 2020)

See MoCo paper note. Key idea: FIFO queue of keys from a momentum encoder ${\theta }_k\leftarrow m{\theta }_k+(1-m){\theta }_q$ — decouples#negatives from batch size. Gradient flows only through the query encoder; the key encoder is detached and updated by EMA.

DINO (Caron 2021) — self-distillation without negatives

See DINO paper note. No explicit negatives: student $g_{{\theta }_s}$ matches a teacher $g_{{\theta }_t}$ (EMA of student) via cross-entropy on softmax outputs. Collapse prevented by centering (subtract running mean from teacher logits) + sharpening (low teacher temperature). Emergent property: attention maps of a ViT trained with DINO produce unsupervised object segmentation.

Source

CS231n 2025 Lec 12 slides ~67–110 (contrastive framework, InfoNCE + MI bound, SimCLR full algorithm + batch-size ablation, MoCo queue + momentum update, MoCo-v2 hybrid, CPC sequence formulation + audio/image results, DINO self-distillation + centering/sharpening). 2026 PDF not published — using 2025 fallback.

Related

• Contrastive learning enables Transfer Learning
• Self-Supervised Learning
• InfoNCE
• SimCLR
• MoCo
• DINO