  Reinforcement learning consists of finding policies that maximize an expected
cumulative long-term reward in a Markov decision process with unknown
transition probabilities and instantaneous rewards. In this paper, we consider
the problem of finding such optimal policies while assuming they are continuous
functions belonging to a reproducing kernel Hilbert space (RKHS). To learn the
optimal policy we introduce a stochastic policy gradient ascent algorithm with
three unique novel features: (i) The stochastic estimates of policy gradients
are unbiased. (ii) The variance of stochastic gradients is reduced by drawing
on ideas from numerical differentiation. (iii) Policy complexity is controlled
using sparse RKHS representations. Novel feature (i) is instrumental in proving
convergence to a stationary point of the expected cumulative reward. Novel
feature (ii) facilitates reasonable convergence times. Novel feature (iii) is a
necessity in practical implementations which we show can be done in a way that
does not eliminate convergence guarantees. Numerical examples in standard
problems illustrate successful learning of policies with low complexity
representations which are close to stationary points of the expected cumulative
reward.
