  Before deploying outputs from foundation models in high-stakes tasks, it is
imperative to ensure that they align with human values. For instance, in
radiology report generation, reports generated by a vision-language model must
align with human evaluations before their use in medical decision-making. This
paper presents Conformal Alignment, a general framework for identifying units
whose outputs meet a user-specified alignment criterion. It is guaranteed that
on average, a prescribed fraction of selected units indeed meet the alignment
criterion, regardless of the foundation model or the data distribution. Given
any pre-trained model and new units with model-generated outputs, Conformal
Alignment leverages a set of reference data with ground-truth alignment status
to train an alignment predictor. It then selects new units whose predicted
alignment scores surpass a data-dependent threshold, certifying their
corresponding outputs as trustworthy. Through applications to question
answering and radiology report generation, we demonstrate that our method is
able to accurately identify units with trustworthy outputs via lightweight
training over a moderate amount of reference data. En route, we investigate the
informativeness of various features in alignment prediction and combine them
with standard models to construct the alignment predictor.
