  Imitation learning is a popular approach for teaching motor skills to robots.
However, most approaches focus on extracting policy parameters from execution
traces alone (i.e., motion trajectories and perceptual data). No adequate
communication channel exists between the human expert and the robot to describe
critical aspects of the task, such as the properties of the target object or
the intended shape of the motion. Motivated by insights into the human teaching
process, we introduce a method for incorporating unstructured natural language
into imitation learning. At training time, the expert can provide
demonstrations along with verbal descriptions in order to describe the
underlying intent (e.g., "go to the large green bowl"). The training process
then interrelates these two modalities to encode the correlations between
language, perception, and motion. The resulting language-conditioned visuomotor
policies can be conditioned at runtime on new human commands and instructions,
which allows for more fine-grained control over the trained policies while also
reducing situational ambiguity. We demonstrate in a set of simulation
experiments how our approach can learn language-conditioned manipulation
policies for a seven-degree-of-freedom robot arm and compare the results to a
variety of alternative methods.
