  Language models (LMs) have become important tools in a variety of
applications, from data processing to the creation of instruction-following
assistants. But despite their advantages, LMs have certain idiosyncratic
limitations such as the problem of `strong priors', where a model learns to
output typical continuations in response to certain, usually local, portions of
the input regardless of any earlier instructions. For example, prompt injection
attacks can induce models to ignore explicit directives. In some cases, larger
models have been shown to be more susceptible to these problems than similar
smaller models, an example of the phenomenon of `inverse scaling'. We develop a
new technique for mitigating the problem of strong priors: we take the original
set of instructions, produce a weakened version of the original prompt that is
even more susceptible to the strong priors problem, and then extrapolate the
continuation away from the weakened prompt. This lets us infer how the model
would continue a hypothetical strengthened set of instructions. Our technique
conceptualises LMs as mixture models which combine a family of data generation
processes, reinforcing the desired elements of the mixture. Our approach works
at inference time, removing any need for retraining. We apply it to eleven
models including GPT-2, GPT-3, Llama 2, and Mistral on four tasks, and find
improvements in 41/44. Across all 44 combinations the median increase in
proportion of tasks completed is 40%.
