  Solving real-world optimal control problems are challenging tasks, as the
complex, high-dimensional system dynamics are usually unrevealed to the
decision maker. It is thus hard to find the optimal control actions
numerically. To deal with such modeling and computation challenges, in this
paper, we integrate Neural Networks with the Pontryagin's Maximum Principle
(PMP), and propose a sample efficient framework NN-PMP-Gradient. The resulting
controller can be implemented for systems with unknown and complex dynamics. By
taking an iterative approach, the proposed framework not only utilizes the
accurate surrogate models parameterized by neural networks, it also efficiently
recovers the optimality conditions along with the optimal action sequences via
PMP conditions. Numerical simulations on Linear Quadratic Regulator, energy
arbitrage of grid-connected lossy battery, control of single pendulum, and two
MuJoCo locomotion tasks demonstrate our proposed NN-PMP-Gradient is a general
and versatile computation tool for finding optimal solutions. And compared with
the widely applied model-free and model-based reinforcement learning (RL)
algorithms, our NN-PMP-Gradient achieves higher sample-efficiency and
performance in terms of control objectives.
