  Despite the general capabilities of large pretrained language models, they
consistently benefit from further adaptation to better achieve desired
behaviors. However, tuning these models has become increasingly
resource-intensive, or impossible when model weights are private. We introduce
proxy-tuning, a lightweight decoding-time algorithm that operates on top of
black-box LMs to achieve the same end as direct tuning, but by accessing only
its predictions over the output vocabulary, not its parameters. Our method
tunes a smaller LM, then applies the difference between the predictions of the
small tuned and untuned LMs to shift the original predictions of the larger
untuned model in the direction of tuning, while retaining the benefits of
larger-scale pretraining. In experiments, when we apply proxy-tuning to
Llama2-70B using proxies of only 7B size, we can close 88% of the gap between
Llama2-70B and its truly-tuned chat version, when evaluated across knowledge,
reasoning, and safety benchmarks. Interestingly, on TruthfulQA, proxy-tuned
models are actually more truthful than directly tuned models, possibly because
decoding-time guidance better retains the model's factual knowledge. We then
demonstrate the generality of proxy-tuning by applying it to domain adaptation
on code, and task-specific finetuning on question-answering and math problems.
Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal
adaptation, increasing its knowledge about recent events. Our work demonstrates
the promise of using small tuned LMs to efficiently customize large,
potentially proprietary LMs through decoding-time guidance.
