RT-1: Robotics Transformer for Real-World Control at Scale

By transferring knowledge from large, diverse, taskagnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small taskspecific datasets to a high level of performance.While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data.We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with highcapacity architectures that can absorb all of the diverse, robotic data.In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties.We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.How would you bring me two sodas?How would you move three cokes to the trash can?How would you throw away two cokes?How would you bring me two different sodas?How would you bring me an apple, a coke, and water bottle?I spilled my coke on the table, how would you throw it away and then bring me something to help clean?I just worked out, can you bring me a drink and a snack to recover?How would you bring me a fruit, a soda, and a bag of chips for lunchTABLE XI: List of SayCan instructions evaluated in Sec.IV-D