  Proximal Policy Optimization (PPO) has been broadly applied to various
domains, including Large Language Model (LLM) optimization and Robotics
learning, etc. However, PPO is limited by a fixed setting for the clipping
bound. Specifically, there is no theoretical proof that the optimal clipping
bound remains consistent throughout the entire training process. Truncating the
ratio of the new and old policies with a unique clipping bound ensures stable
training and can achieve the best training performance. Additionally, previous
research suggests that a fixed clipping bound limits the agent's exploration.
Therefore, researching a dynamical clipping bound to enhance PPO's performance
can be highly beneficial. Different from previous clipping approaches, we
consider increasing the maximum cumulative Return in reinforcement learning
(RL) tasks as the preference of the RL task, and propose a bi-level proximal
policy optimization paradigm, which involves not only optimizing the policy but
also dynamically adjusting the clipping bound to reflect the preference of the
RL tasks to further elevate the training outcomes and stability of PPO. Based
on this bi-level proximal policy optimization paradigm, we introduce a new
algorithm named Preference based Proximal Policy Optimization (Pb-PPO). This
algorithm utilizes a multi-armed bandit algorithm to reflect RL preferences (we
also validate that such approach can be utilized to reflect human preference),
recommending the optimal clipping bound for PPO in each epoch, thereby
achieving more stable and better training outcomes.
