Self-Supervised Learning


From here.

There are 4 categorizations:

  1. Autoregressive generation (predicts future behaviour based on past behaviour, so kinda like an RNN idea?)
    • Ex: Autoregressive Language Modelling (GPT)
    • Images in raster scan (PixelCNN, PixelRNN, iGPT)
  2. Masked generation(We mask a random portion of information and pretend its missing. Model learns to predict this masked information)
    • Masked Language Modelling (BERT)
  3. Innate relationship prediction (?)
    • Image Rotation
  4. Hybrid self-prediction
    • Combines various self-prediction methods, such as VQGAN