  Systems for language-guided human-robot interaction must satisfy two key
desiderata for broad adoption: adaptivity and learning efficiency.
Unfortunately, existing instruction-following agents cannot adapt, lacking the
ability to incorporate online natural language supervision, and even if they
could, require hundreds of demonstrations to learn even simple policies. In
this work, we address these problems by presenting Language-Informed Latent
Actions with Corrections (LILAC), a framework for incorporating and adapting to
natural language corrections - "to the right," or "no, towards the book" -
online, during execution. We explore rich manipulation domains within a shared
autonomy paradigm. Instead of discrete turn-taking between a human and robot,
LILAC splits agency between the human and robot: language is an input to a
learned model that produces a meaningful, low-dimensional control space that
the human can use to guide the robot. Each real-time correction refines the
human's control space, enabling precise, extended behaviors - with the added
benefit of requiring only a handful of demonstrations to learn. We evaluate our
approach via a user study where users work with a Franka Emika Panda
manipulator to complete complex manipulation tasks. Compared to existing
learned baselines covering both open-loop instruction following and single-turn
shared autonomy, we show that our corrections-aware approach obtains higher
task completion rates, and is subjectively preferred by users because of its
reliability, precision, and ease of use.
