  Careful prompt design is critical to the use of large language models in
zero-shot or few-shot learning. As a consequence, there is a growing interest
in automated methods to design optimal prompts. In this work, we propose
Test-time Prompt Editing using Reinforcement learning (TEMPERA). In contrast to
prior prompt generation methods, TEMPERA can efficiently leverage prior
knowledge, is adaptive to different queries and provides an interpretable
prompt for every query. To achieve this, we design a novel action space that
allows flexible editing of the initial prompts covering a wide set of
commonly-used components like instructions, few-shot exemplars, and
verbalizers. The proposed method achieves significant gains compared with
recent SoTA approaches like prompt tuning, AutoPrompt, and RLPrompt, across a
variety of tasks including sentiment analysis, topic classification, natural
language inference, and reading comprehension. Our method achieves 5.33x on
average improvement in sample efficiency when compared to the traditional
fine-tuning methods.
