  Recent efforts to create challenge benchmarks that test the abilities of
natural language understanding models have largely depended on human
annotations. In this work, we introduce the "Break, Perturb, Build" (BPB)
framework for automatic reasoning-oriented perturbation of question-answer
pairs. BPB represents a question by decomposing it into the reasoning steps
that are required to answer it, symbolically perturbs the decomposition, and
then generates new question-answer pairs. We demonstrate the effectiveness of
BPB by creating evaluation sets for three reading comprehension (RC)
benchmarks, generating thousands of high-quality examples without human
intervention. We evaluate a range of RC models on our evaluation sets, which
reveals large performance gaps on generated examples compared to the original
data. Moreover, symbolic perturbations enable fine-grained analysis of the
strengths and limitations of models. Last, augmenting the training data with
examples generated by BPB helps close the performance gaps, without any drop on
the original data distribution.
