  Widespread adoption of self-driving cars will depend not only on their safety
but largely on their ability to interact with human users. Just like human
drivers, self-driving cars will be expected to understand and safely follow
natural-language directions that suddenly alter the pre-planned route according
to user's preference or in presence of ambiguities, particularly in locations
with poor or outdated map coverage. To this end, we propose a language-grounded
driving agent implementing a hierarchical policy using recurrent layers and
gated attention. The hierarchical approach enables us to reason both in terms
of high-level language instructions describing long time horizons and
low-level, complex, continuous state/action spaces required for real-time
control of a self-driving car. We train our policy with conditional imitation
learning from realistic language data collected from human drivers and
navigators. Through quantitative and interactive experiments within the CARLA
framework, we show that our model can successfully interpret language
instructions and follow them safely, even when generalizing to previously
unseen environments. Code and video are available at
https://sites.google.com/view/language-grounded-driving.
