  User alignment is crucial for adapting general-purpose language models (LMs)
to downstream tasks, but human annotations are often not available for all
types of instructions, especially those with customized constraints. We observe
that user instructions typically contain constraints. While assessing response
quality in terms of the whole instruction is often costly, efficiently
evaluating the satisfaction rate of constraints is feasible. We investigate
common constraints in NLP tasks, categorize them into three classes based on
the types of their arguments, and propose a unified framework, ACT (Aligning to
ConsTraints), to automatically produce supervision signals for user alignment
with constraints. Specifically, ACT uses constraint verifiers, which are
typically easy to implement in practice, to compute constraint satisfaction
rate (CSR) of each response. It samples multiple responses for each prompt and
collect preference labels based on their CSR automatically. Subsequently, ACT
adapts the LM to the target task through a ranking-based learning process.
Experiments on fine-grained entity typing, abstractive summarization, and
temporal question answering show that ACT is able to enhance LMs' capability to
adhere to different classes of constraints, thereby improving task performance.
Further experiments show that the constraint-following capabilities are
transferable.
