  We introduce the framework of performative reinforcement learning where the
policy chosen by the learner affects the underlying reward and transition
dynamics of the environment. Following the recent literature on performative
prediction~\cite{Perdomo et. al., 2020}, we introduce the concept of
performatively stable policy. We then consider a regularized version of the
reinforcement learning problem and show that repeatedly optimizing this
objective converges to a performatively stable policy under reasonable
assumptions on the transition dynamics. Our proof utilizes the dual perspective
of the reinforcement learning problem and may be of independent interest in
analyzing the convergence of other algorithms with decision-dependent
environments. We then extend our results for the setting where the learner just
performs gradient ascent steps instead of fully optimizing the objective, and
for the setting where the learner has access to a finite number of trajectories
from the changed environment. For both settings, we leverage the dual
formulation of performative reinforcement learning and establish convergence to
a stable solution. Finally, through extensive experiments on a grid-world
environment, we demonstrate the dependence of convergence on various parameters
e.g. regularization, smoothness, and the number of samples.
