  Most existing theoretical analysis of reinforcement learning (RL) is limited
to the tabular setting or linear models due to the difficulty in dealing with
function approximation in high dimensional space with an uncertain environment.
This work offers a fresh perspective into this challenge by analyzing RL in a
general reproducing kernel Hilbert space (RKHS). We consider a family of Markov
decision processes $\mathcal{M}$ of which the reward functions lie in the unit
ball of an RKHS and transition probabilities lie in a given arbitrary set. We
define a quantity called perturbational complexity by distribution mismatch
$\Delta_{\mathcal{M}}(\epsilon)$ to characterize the complexity of the
admissible state-action distribution space in response to a perturbation in the
RKHS with scale $\epsilon$. We show that $\Delta_{\mathcal{M}}(\epsilon)$ gives
both the lower bound of the error of all possible algorithms and the upper
bound of two specific algorithms (fitted reward and fitted Q-iteration) for the
RL problem. Hence, the decay of $\Delta_\mathcal{M}(\epsilon)$ with respect to
$\epsilon$ measures the difficulty of the RL problem on $\mathcal{M}$. We
further provide some concrete examples and discuss whether
$\Delta_{\mathcal{M}}(\epsilon)$ decays fast or not in these examples. As a
byproduct, we show that when the reward functions lie in a high dimensional
RKHS, even if the transition probability is known and the action space is
finite, it is still possible for RL problems to suffer from the curse of
dimensionality.
