  We study a collaborative scenario where a user not only instructs a system to
complete tasks, but also acts alongside it. This allows the user to adapt to
the system abilities by changing their language or deciding to simply
accomplish some tasks themselves, and requires the system to effectively
recover from errors as the user strategically assigns it new goals. We build a
game environment to study this scenario, and learn to map user instructions to
system actions. We introduce a learning approach focused on recovery from
cascading errors between instructions, and modeling methods to explicitly
reason about instructions with multiple goals. We evaluate with a new
evaluation protocol using recorded interactions and online games with human
users, and observe how users adapt to the system abilities.
