  We introduce new planning and reinforcement learning algorithms for
discounted MDPs that utilize an approximate model of the environment to
accelerate the convergence of the value function. Inspired by the splitting
approach in numerical linear algebra, we introduce Operator Splitting Value
Iteration (OS-VI) for both Policy Evaluation and Control problems. OS-VI
achieves a much faster convergence rate when the model is accurate enough. We
also introduce a sample-based version of the algorithm called OS-Dyna. Unlike
the traditional Dyna architecture, OS-Dyna still converges to the correct value
function in presence of model approximation error.
