  Reinforcement learning (RL) methods often rely on massive exploration data to
search optimal policies, and suffer from poor sampling efficiency. This paper
presents a mixed reinforcement learning (mixed RL) algorithm by simultaneously
using dual representations of environmental dynamics to search the optimal
policy with the purpose of improving both learning accuracy and training speed.
The dual representations indicate the environmental model and the state-action
data: the former can accelerate the learning process of RL, while its inherent
model uncertainty generally leads to worse policy accuracy than the latter,
which comes from direct measurements of states and actions. In the framework
design of the mixed RL, the compensation of the additive stochastic model
uncertainty is embedded inside the policy iteration RL framework by using
explored state-action data via iterative Bayesian estimator (IBE). The optimal
policy is then computed in an iterative way by alternating between policy
evaluation (PEV) and policy improvement (PIM). The convergence of the mixed RL
is proved using the Bellman's principle of optimality, and the recursive
stability of the generated policy is proved via the Lyapunov's direct method.
The effectiveness of the mixed RL is demonstrated by a typical optimal control
problem of stochastic non-affine nonlinear systems (i.e., double lane change
task with an automated vehicle).
