  Recent advancement in combining trajectory optimization with function
approximation (especially neural networks) shows promise in learning complex
control policies for diverse tasks in robot systems. Despite their great
flexibility, the large neural networks for parameterizing control policies
impose significant challenges. The learned neural control policies are often
overcomplex and non-smooth, which can easily cause unexpected or diverging
robot motions. Therefore, they often yield poor generalization performance in
practice. To address this issue, we propose adVErsarially Regularized pOlicy
learNIng guided by trajeCtory optimizAtion (VERONICA) for learning smooth
control policies. Specifically, our proposed approach controls the smoothness
(local Lipschitz continuity) of the neural control policies by stabilizing the
output control with respect to the worst-case perturbation to the input state.
Our experiments on robot manipulation show that our proposed approach not only
improves the sample efficiency of neural policy learning but also enhances the
robustness of the policy against various types of disturbances, including
sensor noise, environmental uncertainty, and model mismatch.
