RT-1: Robotics Transformer for Real-World Control at Scale

My question is, how is it trained on some of the datasets where there are no instruction annotations??

• Then, the instruction is just empty

“It takes in a history of 15 images along with the natural language“.

Related

• RT-2
• RT-H

Citation

(Brohan et al., 2022)

Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N. J., Julian, R. C., Kalashnikov, D., Kuang, Y., … Zitkovich, B. (2022). RT-1: Robotics Transformer for Real-World Control at Scale. https://arxiv.org/abs/2212.06817