  The technique of Reinforcement Learning from Human Feedback (RLHF) is a
commonly employed method to improve pre-trained Language Models (LM), enhancing
their ability to conform to human preferences. Nevertheless, the current
RLHF-based LMs necessitate full retraining each time novel queries or feedback
are introduced, which becomes a challenging task because human preferences can
vary between different domains or tasks. Retraining LMs poses practical
difficulties in many real-world situations due to the significant time and
computational resources required, along with concerns related to data privacy.
To address this limitation, we propose a new method called Continual Optimal
Policy Regularization (COPR), in which we compute the distribution of optimal
policy bypassing the partition function and then regularize the current policy
based on the historically optimal distribution to mitigate Catastrophic
Forgetting (CF). COPR involves a single learning phase and doesn't necessitate
complex reinforcement learning. Importantly, it shares the capability with RLHF
to learn from unlabeled data by maintaining a scoring module, similar to reward
model, making it flexible for continually learning without human feedback. Our
experimental results show that COPR outperforms strong Continuous Learning (CL)
baselines when it comes to consistently aligning with human preferences on
incremental tasks and domains.
