  In this work, we introduce a stochastic maximum principle (SMP) approach for
solving the reinforcement learning problem with the assumption that the
unknowns in the environment can be parameterized based on physics knowledge.
For the development of numerical algorithms, we shall apply an effective online
parameter estimation method as our exploration technique to estimate the
environment parameter during the training procedure, and the exploitation for
the optimal policy will be achieved by an efficient backward action learning
method for policy improvement under the SMP framework. Numerical experiments
will be presented to demonstrate that our SMP approach for reinforcement
learning can produce reliable control policy, and the gradient descent type
optimization in the SMP solver requires less training episodes compared with
the standard dynamic programming principle based methods.
