  Prompt optimization aims to find the best prompt to a large language model
(LLM) for a given task. LLMs have been successfully used to help find and
improve prompt candidates for single-step tasks. However, realistic tasks for
agents are multi-step and introduce new challenges: (1) Prompt content is
likely to be more extensive and complex, making it more difficult for LLMs to
analyze errors, (2) the impact of an individual step is difficult to evaluate,
and (3) different people may have varied preferences about task execution.
While humans struggle to optimize prompts, they are good at providing feedback
about LLM outputs; we therefore introduce a new LLM-driven discrete prompt
optimization framework PROMST that incorporates human-designed feedback rules
to automatically offer direct suggestions for improvement. We also use an extra
learned heuristic model that predicts prompt performance to efficiently sample
from prompt candidates. This approach significantly outperforms both
human-engineered prompts and several other prompt optimization methods across
11 representative multi-step tasks (an average 10.6\%-29.3\% improvement to
current best methods on five LLMs respectively). We believe our work can serve
as a benchmark for automatic prompt optimization for LLM-driven multi-step
tasks. Datasets and Codes are available at
https://github.com/yongchao98/PROMST. Project Page is available at
https://yongchao98.github.io/MIT-REALM-PROMST/.
