  Alignment is a crucial step to enhance the instruction-following and
conversational abilities of language models. Despite many recent work proposing
new algorithms, datasets, and training pipelines, there is a lack of
comprehensive studies measuring the impact of various design choices throughout
the whole training process. We first conduct a rigorous analysis over a
three-stage training pipeline consisting of supervised fine-tuning, offline
preference learning, and online preference learning. We have found that using
techniques like sequence packing, loss masking in SFT, increasing the
preference dataset size in DPO, and online DPO training can significantly
improve the performance of language models. We then train from Gemma-2b-base
and LLama-3-8b-base, and find that our best models exceed the performance of
the official instruct models tuned with closed-source data and algorithms. Our
code and models can be found at
https://github.com/Columbia-NLP-Lab/LionAlignment.
