GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

Project page:

• https://gr2-manipulation.github.io/

Note

Interesting robot foundation model that in the pre-training stage learns to predict images, and during finetuning learns to predict both image and action.

“GR-2 is a language-conditioned GPT-style visual manipulation policy model (Fig. 1).

The training undergoes two stages:

1. video generative pre-training and robot data fine-tuning. During the pre-training stage, we train GR-2 on a curated large-scale video dataset
2. Fine-tune GR-2 on robot data to predict action trajectories and videos in tandem

They use a VQGAN. Why don’t they use a diffusion model?

This is their way of teaching a world model.

Citation

(Cheang et al., 2024)

Cheang, C.-L., Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., Zhang, H., & Zhu, M. (2024). GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation. https://arxiv.org/pdf/2410.06158