  Proximal policy optimization (PPO) is one of the most popular deep
reinforcement learning (RL) methods, achieving state-of-the-art performance
across a wide range of challenging tasks. However, as a model-free RL method,
the success of PPO relies heavily on the effectiveness of its exploratory
policy search. In this paper, we give an in-depth analysis on the exploration
behavior of PPO, and show that PPO is prone to suffer from the risk of lack of
exploration especially under the case of bad initialization, which may lead to
the failure of training or being trapped in bad local optima. To address these
issues, we proposed a novel policy optimization method, named Trust
Region-Guided PPO (TRGPPO), which adaptively adjusts the clipping range within
the trust region. We formally show that this method not only improves the
exploration ability within the trust region but enjoys a better performance
bound compared to the original PPO as well. Extensive experiments verify the
advantage of the proposed method.
