  Generalization remains one of the most important desiderata for robust robot
learning systems. While recently proposed approaches show promise in
generalization to novel objects, semantic concepts, or visual distribution
shifts, generalization to new tasks remains challenging. For example, a
language-conditioned policy trained on pick-and-place tasks will not be able to
generalize to a folding task, even if the arm trajectory of folding is similar
to pick-and-place. Our key insight is that this kind of generalization becomes
feasible if we represent the task through rough trajectory sketches. We propose
a policy conditioning method using such rough trajectory sketches, which we
call RT-Trajectory, that is practical, easy to specify, and allows the policy
to effectively perform new tasks that would otherwise be challenging to
perform. We find that trajectory sketches strike a balance between being
detailed enough to express low-level motion-centric guidance while being coarse
enough to allow the learned policy to interpret the trajectory sketch in the
context of situational visual observations. In addition, we show how trajectory
sketches can provide a useful interface to communicate with robotic policies:
they can be specified through simple human inputs like drawings or videos, or
through automated methods such as modern image-generating or
waypoint-generating methods. We evaluate RT-Trajectory at scale on a variety of
real-world robotic tasks, and find that RT-Trajectory is able to perform a
wider range of tasks compared to language-conditioned and goal-conditioned
policies, when provided the same training data.
