  In this paper, we analyze the behavior of existing techniques and design new
solutions for the problem of one-shot visual imitation. In this setting, an
agent must solve a novel instance of a novel task given just a single visual
demonstration. Our analysis reveals that current methods fall short because of
three errors: the DAgger problem arising from purely offline training, last
centimeter errors in interacting with objects, and mis-fitting to the task
context rather than to the actual task. This motivates the design of our
modular approach where we a) separate out task inference (what to do) from task
execution (how to do it), and b) develop data augmentation and generation
techniques to mitigate mis-fitting. The former allows us to leverage
hand-crafted motor primitives for task execution which side-steps the DAgger
problem and last centimeter errors, while the latter gets the model to focus on
the task rather than the task context. Our model gets 100% and 48% success
rates on two recent benchmarks, improving upon the current state-of-the-art by
absolute 90% and 20% respectively.
