  Controlled generation refers to the problem of creating text that contains
stylistic or semantic attributes of interest. Many approaches reduce this
problem to training a predictor of the desired attribute. For example,
researchers hoping to deploy a large language model to produce non-toxic
content may use a toxicity classifier to filter generated text. In practice,
the generated text to classify, which is determined by user prompts, may come
from a wide range of distributions. In this paper, we show that the performance
of controlled generation may be poor if the distributions of text in response
to user prompts differ from the distribution the predictor was trained on. To
address this problem, we cast controlled generation under distribution shift as
an invariant learning problem: the most effective predictor should be invariant
across multiple text environments. We then discuss a natural solution that
arises from this characterization and propose heuristics for selecting natural
environments. We study this characterization and the proposed method
empirically using both synthetic and real data. Experiments demonstrate both
the challenge of distribution shift in controlled generation and the potential
of invariance methods in this setting.
