  Reinforcement learning (RL) with continuous state and action spaces remains
one of the most challenging problems within the field. Most current learning
methods focus on integral identities such as value functions to derive an
optimal strategy for the learning agent. In this paper, we instead study the
dual form of the original RL formulation to propose the first differential RL
framework that can handle settings with limited training samples and
short-length episodes. Our approach introduces Differential Policy Optimization
(DPO), a pointwise and stage-wise iteration method that optimizes policies
encoded by local-movement operators. We prove a pointwise convergence estimate
for DPO and provide a regret bound comparable with current theoretical works.
Such pointwise estimate ensures that the learned policy matches the optimal
path uniformly across different steps. We then apply DPO to a class of
practical RL problems which search for optimal configurations with Lagrangian
rewards. DPO is easy to implement, scalable, and shows competitive results on
benchmarking experiments against several popular RL methods.
