  There have been many recent advances on provably efficient Reinforcement
Learning (RL) in problems with rich observation spaces. However, all these
works share a strong realizability assumption about the optimal value function
of the true MDP. Such realizability assumptions are often too strong to hold in
practice. In this work, we consider the more realistic setting of agnostic RL
with rich observation spaces and a fixed class of policies $\Pi$ that may not
contain any near-optimal policy. We provide an algorithm for this setting whose
error is bounded in terms of the rank $d$ of the underlying MDP. Specifically,
our algorithm enjoys a sample complexity bound of $\widetilde{O}\left((H^{4d}
K^{3d} \log |\Pi|)/\epsilon^2\right)$ where $H$ is the length of episodes, $K$
is the number of actions and $\epsilon>0$ is the desired sub-optimality. We
also provide a nearly matching lower bound for this agnostic setting that shows
that the exponential dependence on rank is unavoidable, without further
assumptions.
