  The ultimate goals of robotic learning is to acquire a comprehensive and
generalizable robotic system capable of performing both seen skills within the
training distribution and unseen skills in novel environments. Recent progress
in utilizing language models as high-level planners has demonstrated that the
complexity of tasks can be reduced through decomposing them into
primitive-level plans, making it possible to generalize on novel robotic tasks
in a composable manner. Despite the promising future, the community is not yet
adequately prepared for composable generalization agents, particularly due to
the lack of primitive-level real-world robotic datasets. In this paper, we
propose a primitive-level robotic dataset, namely RH20T-P, which contains about
33000 video clips covering 44 diverse and complicated robotic tasks. Each clip
is manually annotated according to a set of meticulously designed primitive
skills, facilitating the future development of composable generalization
agents. To validate the effectiveness of RH20T-P, we also construct a potential
and scalable agent based on RH20T-P, called RA-P. Equipped with two planners
specialized in task decomposition and motion planning, RA-P can adapt to novel
physical skills through composable generalization. Our website and videos can
be found at https://sites.google.com/view/rh20t-primitive/main. Dataset and
code will be made available soon.
