  Large language models (LLMs) have shown impressive success in various
applications. However, these models are often not well aligned with human
intents, which calls for additional treatments on them; that is, the alignment
problem. To make LLMs better follow user instructions, existing alignment
methods primarily focus on further training them. However, the extra training
of LLMs is usually expensive in terms of GPU computing; even worse, some LLMs
are not accessible for user-demanded training, such as GPTs. In this work, we
take a different perspective -- Black-Box Prompt Optimization (BPO) -- to
perform alignments. The idea is to optimize user prompts to suit LLMs' input
understanding, so as to best realize users' intents without updating LLMs'
parameters. BPO leverages human preferences to optimize prompts, thus making it
superior to LLM (e.g., ChatGPT) as a prompt engineer. Moreover, BPO is
model-agnostic, and the empirical results demonstrate that the BPO-aligned
ChatGPT yields a 22% increase in the win rate against its original version and
10% for GPT-4. Notably, the BPO-aligned LLMs can outperform the same models
aligned by PPO and DPO, and it also brings additional performance gains when
combining BPO with PPO or DPO. Code and datasets are released at
https://github.com/thu-coai/BPO.
