  Discrete prompts have been used for fine-tuning Pre-trained Language Models
for diverse NLP tasks. In particular, automatic methods that generate discrete
prompts from a small set of training instances have reported superior
performance. However, a closer look at the learnt prompts reveals that they
contain noisy and counter-intuitive lexical constructs that would not be
encountered in manually-written prompts. This raises an important yet
understudied question regarding the robustness of automatically learnt discrete
prompts when used in downstream tasks. To address this question, we conduct a
systematic study of the robustness of discrete prompts by applying carefully
designed perturbations into an application using AutoPrompt and then measure
their performance in two Natural Language Inference (NLI) datasets. Our
experimental results show that although the discrete prompt-based method
remains relatively robust against perturbations to NLI inputs, they are highly
sensitive to other types of perturbations such as shuffling and deletion of
prompt tokens. Moreover, they generalize poorly across different NLI datasets.
We hope our findings will inspire future work on robust discrete prompt
learning.
