  Ensuring alignment with human preferences is a crucial characteristic of
large language models (LLMs). Presently, the primary alignment methods, RLHF
and DPO, require extensive human annotation, which is expensive despite their
efficacy. The significant expenses associated with current alignment techniques
motivate researchers to investigate the development of annotation-free
alignment training methods. In pursuit of improved alignment without relying on
external annotation, we introduce Latent Distance Guided Alignment Training
(LD-Align). This approach seeks to align the model with a high-quality
supervised fine-tune dataset using guidance from a latent space. The latent
space is generated through sample reconstruction, akin to auto-encoding.
Consequently, we utilize the distance between sample pairs in the latent space
to guide DPO-based alignment training. Extensive experimentation and evaluation
show the efficacy of our proposed method in achieving notable alignment.
