  We develop a model selection approach to tackle reinforcement learning with
adversarial corruption in both transition and reward. For finite-horizon
tabular MDPs, without prior knowledge on the total amount of corruption, our
algorithm achieves a regret bound of
$\widetilde{\mathcal{O}}(\min\{\frac{1}{\Delta}, \sqrt{T}\}+C)$ where $T$ is
the number of episodes, $C$ is the total amount of corruption, and $\Delta$ is
the reward gap between the best and the second-best policy. This is the first
worst-case optimal bound achieved without knowledge of $C$, improving previous
results of Lykouris et al. (2021); Chen et al. (2021); Wu et al. (2021). For
finite-horizon linear MDPs, we develop a computationally efficient algorithm
with a regret bound of $\widetilde{\mathcal{O}}(\sqrt{(1+C)T})$, and another
computationally inefficient one with $\widetilde{\mathcal{O}}(\sqrt{T}+C)$,
improving the result of Lykouris et al. (2021) and answering an open question
by Zhang et al. (2021b). Finally, our model selection framework can be easily
applied to other settings including linear bandits, linear contextual bandits,
and MDPs with general function approximation, leading to several improved or
new results.
