  Natural language and images are commonly used as goal representations in
goal-conditioned imitation learning (IL). However, natural language can be
ambiguous and images can be over-specified. In this work, we propose hand-drawn
sketches as a modality for goal specification in visual imitation learning.
Sketches are easy for users to provide on the fly like language, but similar to
images they can also help a downstream policy to be spatially-aware and even go
beyond images to disambiguate task-relevant from task-irrelevant objects. We
present RT-Sketch, a goal-conditioned policy for manipulation that takes a
hand-drawn sketch of the desired scene as input, and outputs actions. We train
RT-Sketch on a dataset of paired trajectories and corresponding synthetically
generated goal sketches. We evaluate this approach on six manipulation skills
involving tabletop object rearrangements on an articulated countertop.
Experimentally we find that RT-Sketch is able to perform on a similar level to
image or language-conditioned agents in straightforward settings, while
achieving greater robustness when language goals are ambiguous or visual
distractors are present. Additionally, we show that RT-Sketch has the capacity
to interpret and act upon sketches with varied levels of specificity, ranging
from minimal line drawings to detailed, colored drawings. For supplementary
material and videos, please refer to our website: http://rt-sketch.github.io.
