  The Reinforcement Learning from Human Feedback (RLHF) plays a pivotal role in
shaping the impact of large language models (LLMs), contributing significantly
to controlling output toxicity and selecting output styles, particularly as
LLMs often harbor misleading content, highlighting the urgency to align them
with human values for secure AI systems. The RLHF, characterized by complexity,
instability, and sensitivity to hyperparameters, makes the evaluation of the
reward model for complex tasks challenging, thereby further complicating the
use of Proximal Policy Optimization (PPO). In this paper, we introduce a simple
task designed to employ Gloden as a reward model that validates the
effectiveness of PPO and inspires it, primarily explaining the task of
utilizing PPO to manipulate the tokenizer length of the output generated by the
model. Experiments confirm that PPO is not only effective in manipulating the
output tokenizer length to a certain extent in this type of task but also
exhibits facilitated training once the influence of the reward model effect is
excluded, making it an exciting development.
