  We have recently witnessed a number of impressive results on hard
mathematical reasoning problems with language models. At the same time, the
robustness of these models has also been called into question; recent works
have shown that models can rely on shallow patterns in the problem description
when generating a solution. Building on the idea of behavioral testing, we
propose a novel framework, which pins down the causal effect of various factors
in the input, e.g., the surface form of the problem text, the operands, and
math operators on the output solution. By grounding the behavioral analysis in
a causal graph describing an intuitive reasoning process, we study the behavior
of language models in terms of robustness and sensitivity to direct
interventions in the input space. We apply our framework on a test bed of math
word problems. Our analysis shows that robustness does not appear to
continuously improve as a function of size, but the GPT-3 Davinci models (175B)
achieve a dramatic improvement in both robustness and sensitivity compared to
all other GPT variants.
