  Motivated by the success of reinforcement learning (RL) for discrete-time
tasks such as AlphaGo and Atari games, there has been a recent surge of
interest in using RL for continuous-time control of physical systems (cf. many
challenging tasks in OpenAI Gym and DeepMind Control Suite). Since
discretization of time is susceptible to error, it is methodologically more
desirable to handle the system dynamics directly in continuous time. However,
very few techniques exist for continuous-time RL and they lack flexibility in
value function approximation. In this paper, we propose a novel framework for
model-based continuous-time value function approximation in reproducing kernel
Hilbert spaces. The resulting framework is so flexible that it can accommodate
any kind of kernel-based approach, such as Gaussian processes and kernel
adaptive filters, and it allows us to handle uncertainties and nonstationarity
without prior knowledge about the environment or what basis functions to
employ. We demonstrate the validity of the presented framework through
experiments.
