  When robots interact with humans in homes, roads, or factories the human's
behavior often changes in response to the robot. Non-stationary humans are
challenging for robot learners: actions the robot has learned to coordinate
with the original human may fail after the human adapts to the robot. In this
paper we introduce an algorithmic formalism that enables robots (i.e., ego
agents) to co-adapt alongside dynamic humans (i.e., other agents) using only
the robot's low-level states, actions, and rewards. A core challenge is that
humans not only react to the robot's behavior, but the way in which humans
react inevitably changes both over time and between users. To deal with this
challenge, our insight is that -- instead of building an exact model of the
human -- robots can learn and reason over high-level representations of the
human's policy and policy dynamics. Applying this insight we develop RILI:
Robustly Influencing Latent Intent. RILI first embeds low-level robot
observations into predictions of the human's latent strategy and strategy
dynamics. Next, RILI harnesses these predictions to select actions that
influence the adaptive human towards advantageous, high reward behaviors over
repeated interactions. We demonstrate that -- given RILI's measured performance
with users sampled from an underlying distribution -- we can probabilistically
bound RILI's expected performance across new humans sampled from the same
distribution. Our simulated experiments compare RILI to state-of-the-art
representation and reinforcement learning baselines, and show that RILI better
learns to coordinate with imperfect, noisy, and time-varying agents. Finally,
we conduct two user studies where RILI co-adapts alongside actual humans in a
game of tag and a tower-building task. See videos of our user studies here:
https://youtu.be/WYGO5amDXbQ
