  We propose a principled kernel-based policy iteration algorithm to solve the
continuous-state Markov Decision Processes (MDPs). In contrast to most
decision-theoretic planning frameworks, which assume fully known state
transition models, we design a method that eliminates such a strong assumption,
which is oftentimes extremely difficult to engineer in reality. To achieve
this, we first apply the second-order Taylor expansion of the value function.
The Bellman optimality equation is then approximated by a partial differential
equation, which only relies on the first and second moments of the transition
model. By combining the kernel representation of value function, we then design
an efficient policy iteration algorithm whose policy evaluation step can be
represented as a linear system of equations characterized by a finite set of
supporting states. We have validated the proposed method through extensive
simulations in both simplified and realistic planning scenarios, and the
experiments show that our proposed approach leads to a much superior
performance over several baseline methods.
