  Reinforcement learning (RL) has emerged as a powerful paradigm for
fine-tuning Large Language Models (LLMs) for text generation. In particular,
recent LLMs such as ChatGPT and GPT-4 can engage in fluent conversations with
users after finetuning with RL. Capitalizing on key properties of text
generation, we seek to investigate RL algorithms beyond general purpose
algorithms like Proximal Policy Optimization (PPO). In particular, we extend RL
algorithms to allow them to interact with a dynamic black-box guide LLM and
propose RL with guided feedback (RLGF), a suite of RL algorithms for LLM
fine-tuning. We provide two ways for the guide LLM to interact with the LLM to
be optimized for maximizing rewards. The guide LLM can generate text which
serves as additional starting states for the RL optimization procedure. The
guide LLM can also be used to complete the partial sentences generated by the
LLM that is being optimized, treating the guide LLM as an expert to imitate and
surpass eventually. We experiment on the IMDB positive sentiment, CommonGen,
and TL;DR summarization tasks. We show that our RL algorithms achieve higher
performance than supervised learning (SL) and the RL baseline PPO,
demonstrating the benefit of interaction with the guide LLM. On both CommonGen
and TL;DR, we not only outperform our SL baselines but also improve upon PPO
across a variety of metrics beyond the one we optimized for. Our code can be
found at https://github.com/Cornell-RL/tril.
