  This paper presents a constrained adaptive dynamic programming (CADP)
algorithm to solve general nonlinear nonaffine optimal control problems with
known dynamics. Unlike previous ADP algorithms, it can directly deal with
problems with state constraints. Firstly, a constrained generalized policy
iteration (CGPI) framework is developed to handle state constraints by
transforming the traditional policy improvement process into a constrained
policy optimization problem. Next, we propose an actor-critic variant of CGPI,
called CADP, in which both policy and value functions are approximated by
multi-layer neural networks to directly map the system states to control inputs
and value function, respectively. CADP linearizes the constrained optimization
problem locally into a quadratically constrained linear programming problem,
and then obtains the optimal update of the policy network by solving its dual
problem. A trust region constraint is added to prevent excessive policy update,
thus ensuring linearization accuracy. We determine the feasibility of the
policy optimization problem by calculating the minimum trust region boundary
and update the policy using two recovery rules when infeasible. The vehicle
control problem in the path-tracking task is used to demonstrate the
effectiveness of this proposed method.
