  This article presents a constrained policy optimization approach for the
optimal control of systems under nonstationary uncertainties. We introduce an
assumption that we call Markov embeddability that allows us to cast the
stochastic optimal control problem as a policy optimization problem over the
augmented state space. Then, the infinite-dimensional policy optimization
problem is approximated as a finite-dimensional nonlinear program by applying
function approximation, deterministic sampling, and temporal truncation. The
approximated problem is solved by using automatic differentiation and
condensed-space interior-point methods. We formulate several conceptual and
practical open questions regarding the asymptotic exactness of the
approximation and the solution strategies for the approximated problem. As a
proof of concept, we provide a numerical example demonstrating the performance
of the control policy obtained by the proposed method.
