  Learning from human preferences is crucial for language models (LMs) to
effectively cater to human needs and societal values. Previous research has
made notable progress by leveraging human feedback to follow instructions.
However, these approaches rely primarily on online learning techniques like
Proximal Policy Optimization (PPO), which have been proven unstable and
challenging to tune for language models. Moreover, PPO requires complex
distributed system implementation, hindering the efficiency of large-scale
distributed training. In this study, we propose an offline learning from human
feedback framework to align LMs without interacting with environments.
Specifically, we explore filtering alignment (FA), reward-weighted regression
(RWR), and conditional alignment (CA) to align language models to human
preferences. By employing a loss function similar to supervised fine-tuning,
our methods ensure more stable model training than PPO with a simple machine
learning system~(MLSys) and much fewer (around 9\%) computing resources.
Experimental results demonstrate that conditional alignment outperforms other
offline alignment methods and is comparable to PPO.
