  When interacting with people, AI agents do not just influence the state of
the world -- they also influence the actions people take in response to the
agent, and even their underlying intentions and strategies. Accounting for and
leveraging this influence has mostly been studied in settings where it is
sufficient to assume that human behavior is near-optimal: competitive games, or
general-sum settings like autonomous driving alongside human drivers. Instead,
we focus on influence in settings where there is a need to capture human
suboptimality. For instance, imagine a collaborative task in which, due either
to cognitive biases or lack of information, people do not perform very well --
how could an agent influence them towards more optimal behavior? Assuming
near-optimal human behavior will not work here, and so the agent needs to learn
from real human data. But experimenting online with humans is potentially
unsafe, and creating a high-fidelity simulator of the environment is often
impractical. Hence, we focus on learning from an offline dataset of human-human
interactions. Our observation is that offline reinforcement learning (RL) can
learn to effectively influence suboptimal humans by extending and combining
elements of observed human-human behavior. We demonstrate that offline RL can
solve two challenges with effective influence. First, we show that by learning
from a dataset of suboptimal human-human interaction on a variety of tasks --
none of which contains examples of successful influence -- an agent can learn
influence strategies to steer humans towards better performance even on new
tasks. Second, we show that by also modeling and conditioning on human
behavior, offline RL can learn to affect not just the human's actions but also
their underlying strategy, and adapt to changes in their strategy.
