  In this paper, we consider the problem of online learning of Markov decision
processes (MDPs) with very large state spaces. Under the assumptions of
realizable function approximation and low Bellman ranks, we develop an online
learning algorithm that learns the optimal value function while at the same
time achieving very low cumulative regret during the learning process. Our
learning algorithm, Adaptive Value-function Elimination (AVE), is inspired by
the policy elimination algorithm proposed in (Jiang et al., 2017), known as
OLIVE. One of our key technical contributions in AVE is to formulate the
elimination steps in OLIVE as contextual bandit problems. This technique
enables us to apply the active elimination and expert weighting methods from
(Dudik et al., 2011), instead of the random action exploration scheme used in
the original OLIVE algorithm, for more efficient exploration and better control
of the regret incurred in each policy elimination step. To the best of our
knowledge, this is the first $\sqrt{n}$-regret result for reinforcement
learning in stochastic MDPs with general value function approximation.
