  In classic instruction following, language like "I'd like the JetBlue flight"
maps to actions (e.g., selecting that flight). However, language also conveys
information about a user's underlying reward function (e.g., a general
preference for JetBlue), which can allow a model to carry out desirable actions
in new contexts. We present a model that infers rewards from language
pragmatically: reasoning about how speakers choose utterances not only to
elicit desired actions, but also to reveal information about their preferences.
On a new interactive flight-booking task with natural language, our model more
accurately infers rewards and predicts optimal actions in unseen environments,
in comparison to past work that first maps language to actions (instruction
following) and then maps actions to rewards (inverse reinforcement learning).
