  Aligning language models (LMs) with preferences is an important problem in
natural language generation. A key challenge is that preferences are typically
provided at the *sequence level* while LM training and generation both occur at
the *token level*. There is, therefore, a *granularity mismatch* between the
preference and the LM training losses, which may complicate the learning
problem. In this paper, we address this issue by developing an alternate
training process, where we iterate between grounding the sequence-level
preference into token-level training guidance, and improving the LM with the
learned guidance. For guidance learning, we design a framework that extends the
pairwise-preference learning in imitation learning to both variable-length LM
generation and the utilization of the preference among multiple generations.
For LM training, based on the amount of supervised data, we present two
*minimalist* learning objectives that utilize the learned guidance. In
experiments, our method performs competitively on two distinct representative
LM tasks -- discrete-prompt generation and text summarization.
