  Recently, ChatGPT or InstructGPT like large language models (LLM) has made a
significant impact in the AI world. Many works have attempted to reproduce the
complex InstructGPT's training pipeline, namely Reinforcement Learning with
Human Feedback (RLHF). However, the mainstream distributed RLHF training
methods typically adopt a fixed model placement strategy, referred to as the
Flattening strategy. This strategy treats all four interdependent models
involved in RLHF as a single entity, distributing them across all devices and
applying parallelism techniques designed for a single model, regardless of the
different workloads inherent to each model. As a result, this strategy
exacerbates the generation bottlenecks in the RLHF training and degrades the
overall training efficiency. To address these issues, we propose an adaptive
model placement framework that offers two flexible model placement strategies.
The Interleaving strategy helps reduce memory redundancy and communication
costs of RLHF training by placing models without dependencies on exclusive
devices with careful orchestration. On the other hand, the Separation strategy
improves the throughput of model training by separating the training and
inference runtime of the RLHF pipeline with additional shadow models.
Furthermore, our framework provides a simple user interface and allows for the
agile allocation of models across devices in a fine-grained manner for various
training scenarios, involving models of varying sizes and devices of different
scales. Extensive experiments have demonstrated that our Interleaving and
Separation strategies can achieve notable improvements up to 11X, compared to
the current SOTA approaches. The results highlight the effectiveness and
adaptability of our approaches in accelerating the training of distributed
RLHF.
