  We introduce Directional Stimulus Prompting, a novel framework for guiding
black-box large language models (LLMs) toward specific desired outputs. Instead
of directly adjusting LLMs, our method employs a small tunable policy model
(e.g., T5) to generate an auxiliary directional stimulus prompt for each input
instance. These directional stimulus prompts act as nuanced, instance-specific
hints and clues to guide LLMs in generating desired outcomes, such as including
specific keywords in the generated summary. Our approach sidesteps the
challenges of direct LLM tuning by optimizing the policy model to explore
directional stimulus prompts that align LLMs with desired behaviors. The policy
model can be optimized through 1) supervised fine-tuning using labeled data and
2) reinforcement learning from offline or online rewards based on the LLM's
output. We assess our method across summarization, dialogue response
generation, and chain-of-thought reasoning tasks. Our experiments demonstrate
that the framework consistently improves LLMs' (e.g., ChatGPT, Codex,
InstructGPT) performance on these supervised tasks using minimal labeled data.
Notably, using just 80 dialogues on the MultiWOZ dataset, our approach enhances
ChatGPT's performance by an impressive 41.4%, matching or surpassing some fully
supervised start-of-the-art models. Additionally, the instance-specific
chain-of-thought prompt generated by our approach improves InstructGPT's
reasoning accuracy compared to human-crafted or automatically generated
prompts. The code and data are publicly available at
\url{https://github.com/Leezekun/Directional-Stimulus-Prompting}.
