  We consider reinforcement learning (RL) in episodic Markov decision processes
(MDPs) with linear function approximation under drifting environment.
Specifically, both the reward and state transition functions can evolve over
time but their total variations do not exceed a $\textit{variation budget}$. We
first develop $\texttt{LSVI-UCB-Restart}$ algorithm, an optimistic modification
of least-squares value iteration with periodic restart, and bound its dynamic
regret when variation budgets are known. Then we propose a parameter-free
algorithm $\texttt{Ada-LSVI-UCB-Restart}$ that extends to unknown variation
budgets. We also derive the first minimax dynamic regret lower bound for
nonstationary linear MDPs and as a byproduct establish a minimax regret lower
bound for linear MDPs unsolved by Jin et al. (2020). Finally, we provide
numerical experiments to demonstrate the effectiveness of our proposed
algorithms.
