  We present Skill Transformer, an approach for solving long-horizon robotic
tasks by combining conditional sequence modeling and skill modularity.
Conditioned on egocentric and proprioceptive observations of a robot, Skill
Transformer is trained end-to-end to predict both a high-level skill (e.g.,
navigation, picking, placing), and a whole-body low-level action (e.g., base
and arm motion), using a transformer architecture and demonstration
trajectories that solve the full task. It retains the composability and
modularity of the overall task through a skill predictor module while reasoning
about low-level actions and avoiding hand-off errors, common in modular
approaches. We test Skill Transformer on an embodied rearrangement benchmark
and find it performs robust task planning and low-level control in new
scenarios, achieving a 2.5x higher success rate than baselines in hard
rearrangement problems.
