  The relationship between the quality of a string and its probability
$p(\boldsymbol{y})$ under a language model has been influential in the
development of techniques to build good text generation systems. For example,
several decoding algorithms have been motivated to manipulate
$p(\boldsymbol{y})$ to produce higher-quality text. In this work, we examine
the probability--quality relationship in language models explicitly aligned to
human preferences, e.g., through Reinforcement Learning through Human Feedback
(RLHF). We find that, given a general language model and its aligned version,
for corpora sampled from an aligned language model, there exists a trade-off
between the average reward and average log-likelihood of the strings under the
general language model. We provide a formal treatment of this issue and
demonstrate how a choice of sampling adaptor allows for a selection of how much
likelihood we exchange for the reward.
