  Direct policy search has achieved great empirical success in reinforcement
learning. Recently, there has been increasing interest in studying its
theoretical properties for continuous control, and fruitful results have been
established for linear quadratic regulator (LQR) and linear quadratic Gaussian
(LQG) control that are smooth and nonconvex. In this paper, we consider the
standard $H_\infty$ robust control for output feedback systems and investigate
the global optimality of direct policy search. Unlike LQR or LQG, the
$H_\infty$ cost function is nonsmooth in the policy space. Despite the lack of
smoothness and convexity, our main result shows that for a class of
non-degenerated stabilizing controllers, all Clarke stationary points of
$H_\infty$ robust control are globally optimal and there is no spurious local
minimum. Our proof technique is motivated by the idea of differentiable convex
liftings (DCL), and we extend DCL to analyze the nonsmooth and nonconvex
$H_\infty$ robust control via convex reformulation. Our result sheds some light
on the analysis of direct policy search for solving nonsmooth and nonconvex
robust control problems.
