Self-Supervised Learning

Self-Prediction

From here.

There are 4 categorizations:

1. Autoregressive generation (predicts future behaviour based on past behaviour, so kinda like an RNN idea?)
   • Ex: Autoregressive Language Modelling (GPT)
   • Images in raster scan (PixelCNN, PixelRNN, iGPT)
2. Masked generation(We mask a random portion of information and pretend its missing. Model learns to predict this masked information)
   • Masked Language Modelling (BERT)
3. Innate relationship prediction (?)
   • Image Rotation
4. Hybrid self-prediction
   • Combines various self-prediction methods, such as VQGAN