Machine Learning

Representation Learning

Representation learning learns useful feature representations of data, often in an unsupervised or self-supervised way, so that they can be reused for downstream tasks (e.g. classification, clustering).

Essentially the latent space that this data gets encoded in should capture some useful representation.

Examples:

• Autoencoders
• Variational Autoencoders (VAE)
• Contrastive Learning
• Masked modeling (e.g., BERT, MAE)

For learning images via contrastive methods:

• SimCLR
• MoCo
• CLIP (though this is for image-text pairs)

Learning without contrastive methods:

• BYOL (no negative examples)
• DINO (no negative examples too)

Via generative-methods

• MAE, VideoMAE

You also have these foundation models

• PaLME An Embodied Multimodal Language Model
• Flamingo a Visual Language Model for FewShot Learning
• SAM

Those that use a contrastive loss in SimCLR and CLIP don’t suffer as much from mode collapse, because the negative examples serve as a regularization term.

Relationship with generative models?

Representation learning and generative models often overlap, but they are not the same thing.

• Many generative models naturally learn useful representations as a by-product.