  The prevailing approach to aligning Large Language Models (LLMs) typically
relies on human or AI feedback and assumes access to specific types of
preference datasets. In our work, we question the efficacy of such datasets and
explore various scenarios where alignment with expert demonstrations proves
more realistic. We build a sequential decision-making framework to formulate
the problem of aligning LLMs using demonstration datasets. Drawing insights
from inverse reinforcement learning and imitation learning, we introduce
various approaches for divergence minimization in the LLM alignment tasks. Our
analysis highlights the mass-covering and mode-seeking behaviors of these
different approaches. Inclusively, we examine the pros and cons of the
classical supervised fine-tuning method, elaborating on scenarios where
different methods shine.
