  We study episodic reinforcement learning (RL) in non-stationary linear kernel
Markov decision processes (MDPs). In this setting, both the reward function and
the transition kernel are linear with respect to the given feature maps and are
allowed to vary over time, as long as their respective parameter variations do
not exceed certain variation budgets. We propose the
$\underline{\text{p}}$eriodically $\underline{\text{r}}$estarted
$\underline{\text{o}}$ptimistic $\underline{\text{p}}$olicy
$\underline{\text{o}}$ptimization algorithm (PROPO), which is an optimistic
policy optimization algorithm with linear function approximation. PROPO
features two mechanisms: sliding-window-based policy evaluation and
periodic-restart-based policy improvement, which are tailored for policy
optimization in a non-stationary environment. In addition, only utilizing the
technique of sliding window, we propose a value-iteration algorithm. We
establish dynamic upper bounds for the proposed methods and a matching minimax
lower bound which shows the (near-) optimality of the proposed methods. To our
best knowledge, PROPO is the first provably efficient policy optimization
algorithm that handles non-stationarity.
