  Optimal control problem is typically solved by first finding the value
function through Hamilton-Jacobi equation (HJE) and then taking the minimizer
of the Hamiltonian to obtain the control. In this work, instead of focusing on
the value function, we propose a new formulation for the gradient of the value
function (value-gradient) as a decoupled system of partial differential
equations in the context of continuous-time deterministic discounted optimal
control problem. We develop an efficient iterative scheme for this system of
equations in parallel by utilizing the properties that they share the same
characteristic curves as the HJE for the value function. For the theoretical
part, we prove that this iterative scheme converges linearly in $L_\alpha^2$
sense for some suitable exponent $\alpha$ in a weight function. For the
numerical method, we combine characteristic line method with machine learning
techniques. Specifically, we generate multiple characteristic curves at each
policy iteration from an ensemble of initial states, and compute both the value
function and its gradient simultaneously on each curve as the labelled data.
Then supervised machine learning is applied to minimize the weighted squared
loss for both the value function and its gradients. Experimental results
demonstrate that this new method not only significantly increases the accuracy
but also improves the efficiency and robustness of the numerical estimates,
particularly with less amount of characteristics data or fewer training steps.
