Bootstrap your own latent: A new approach to self-supervised Learning (BYOL)

Seems like quite an important paper that V-JEPA mentions for preventing mode collapse.

$${\mathcal{L}}_{BYOL}=\Vert {\mathbf{q}}_{\theta }({\mathbf{z}}_1)-\text{sg}({\mathbf{z}}_2^{\prime }){\Vert }_2^2+\Vert {\mathbf{q}}_{\theta }({\mathbf{z}}_2)-\text{sg}({\mathbf{z}}_1^{\prime }){\Vert }_2^2$$

where:

• ${\mathbf{z}}_1=f_{\theta }(v_1)$, ${\mathbf{z}}_2=f_{\theta }(v_2)$ are representations from the online encoder $f_{\theta }$ given two augmentations $v_1,v_2$ of the same image,
• ${\mathbf{z}}_1^{\prime }=f_{\xi }^{\prime }(v_1)$, ${\mathbf{z}}_2^{\prime }=f_{\xi }^{\prime }(v_2)$ are representations from the target encoder $f_{\xi }^{\prime }$ (parameters $\xi$ updated via EMA),
• ${\mathbf{q}}_{\theta }$ is the predictor head on the online branch,
• $\text{sg}(\cdot )$ means stop-gradient (no gradients flow into the target encoder),
• $\Vert \cdot {\Vert }_2^2$ is the squared ${\ell }_2$ distance (MSE).

Intuition

Each online view predicts the target’s representation of the other view. No negatives are involved, collapse is prevented by stop-grad + predictor + momentum target.

Related

• SimCLR

Citation

(Grill et al., 2020)

Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., & Valko, M. (2020). Bootstrap your own latent: A new approach to self-supervised Learning (BYOL). https://arxiv.org/pdf/2006.07733