  AI model alignment is crucial due to inadvertent biases in training data and
the underspecified pipeline in modern machine learning, where numerous models
with excellent test set metrics can be produced, yet they may not meet end-user
requirements. Recent advances demonstrate that post-training model alignment
via human feedback can address some of these challenges. However, these methods
are often confined to settings (such as generative AI) where humans can
interpret model outputs and provide feedback. In traditional non-generative
settings, where model outputs are numerical values or classes, detecting
misalignment through single-sample outputs is highly challenging.
  In this paper we consider an alternative strategy. We propose interpreting
model alignment through property testing, defining an aligned model $f$ as one
belonging to a subset $\mathcal{P}$ of functions that exhibit specific desired
behaviors. We focus on post-processing a pre-trained model $f$ to better align
with $\mathcal{P}$ using conformal risk control. Specifically, we develop a
general procedure for converting queries for a given property $\mathcal{P}$ to
a collection of loss functions suitable for use in a conformal risk control
algorithm. We prove a probabilistic guarantee that the resulting conformal
interval around $f$ contains a function approximately satisfying $\mathcal{P}$.
  Given the capabilities of modern AI models with extensive parameters and
training data, one might assume alignment issues will resolve naturally.
However, increasing training data or parameters in a random feature model
doesn't eliminate the need for alignment techniques when pre-training data is
biased. We demonstrate our alignment methodology on supervised learning
datasets for properties like monotonicity and concavity. Our flexible procedure
can be applied to various desired properties.
