  We study online reinforcement learning (RL) in non-stationary environments,
where a time-varying exogenous context process affects the environment
dynamics. Online RL is challenging in such environments due to "catastrophic
forgetting" (CF). The agent tends to forget prior knowledge as it trains on new
experiences. Prior approaches to mitigate this issue assume task labels (which
are often not available in practice) or use off-policy methods that suffer from
instability and poor performance.
  We present Locally Constrained Policy Optimization (LCPO), an online RL
approach that combats CF by anchoring policy outputs on old experiences while
optimizing the return on current experiences. To perform this anchoring, LCPO
locally constrains policy optimization using samples from experiences that lie
outside of the current context distribution. We evaluate LCPO in Mujoco,
classic control and computer systems environments with a variety of synthetic
and real context traces, and find that it outperforms state-of-the-art
on-policy and off-policy RL methods in the non-stationary setting, while
achieving results on-par with an "oracle" agent trained offline across all
context traces.
