  Proximal Policy Optimization algorithm employing a clipped surrogate
objective (PPO-Clip) is a prominent exemplar of the policy optimization
methods. However, despite its remarkable empirical success, PPO-Clip lacks
theoretical substantiation to date. In this paper, we contribute to the field
by establishing the first global convergence results of a PPO-Clip variant in
both tabular and neural function approximation settings. Our findings highlight
the $O(1/\sqrt{T})$ min-iterate convergence rate specifically in the context of
neural function approximation. We tackle the inherent challenges in analyzing
PPO-Clip through three central concepts: (i) We introduce a generalized version
of the PPO-Clip objective, illuminated by its connection with the hinge loss.
(ii) Employing entropic mirror descent, we establish asymptotic convergence for
tabular PPO-Clip with direct policy parameterization. (iii) Inspired by the
tabular analysis, we streamline convergence analysis by introducing a two-step
policy improvement approach. This decouples policy search from complex neural
policy parameterization using a regression-based update scheme. Furthermore, we
gain deeper insights into the efficacy of PPO-Clip by interpreting these
generalized objectives. Our theoretical findings also mark the first
characterization of the influence of the clipping mechanism on PPO-Clip
convergence. Importantly, the clipping range affects only the pre-constant of
the convergence rate.
