  Masked language models like BERT can perform text classification in a
zero-shot fashion by reformulating downstream tasks as text infilling. However,
this approach is highly sensitive to the template used to prompt the model, yet
practitioners are blind when designing them in strict zero-shot settings. In
this paper, we propose an alternative mining-based approach for zero-shot
learning. Instead of prompting language models, we use regular expressions to
mine labeled examples from unlabeled corpora, which can optionally be filtered
through prompting, and used to finetune a pretrained model. Our method is more
flexible and interpretable than prompting, and outperforms it on a wide range
of tasks when using comparable templates. Our results suggest that the success
of prompting can partly be explained by the model being exposed to similar
examples during pretraining, which can be directly retrieved through regular
expressions.
