  We study the sample complexity of model-based reinforcement learning
(henceforth RL) in general contextual decision processes that require strategic
exploration to find a near-optimal policy. We design new algorithms for RL with
a generic model class and analyze their statistical properties. Our algorithms
have sample complexity governed by a new structural parameter called the
witness rank, which we show to be small in several settings of interest,
including factored MDPs. We also show that the witness rank is never larger
than the recently proposed Bellman rank parameter governing the sample
complexity of the model-free algorithm OLIVE (Jiang et al., 2017), the only
other provably sample-efficient algorithm for global exploration at this level
of generality. Focusing on the special case of factored MDPs, we prove an
exponential lower bound for a general class of model-free approaches, including
OLIVE, which, when combined with our algorithmic results, demonstrates
exponential separation between model-based and model-free RL in some
rich-observation settings.
