  This paper proposes a methodology for generating and perturbing detailed
derivations of equations at scale, aided by a symbolic engine, to evaluate the
generalisability of Transformers to out-of-distribution mathematical reasoning
problems. Instantiating the framework in the context of sequence classification
tasks, we compare the capabilities of GPT-4, GPT-3.5, and a canon of fine-tuned
BERT models, exploring the relationship between specific operators and
generalisation failure via the perturbation of reasoning aspects such as
symmetry and variable surface forms. Surprisingly, our empirical evaluation
reveals that the average in-distribution performance of fine-tuned models
surpasses GPT-3.5, and rivals GPT-4. However, perturbations to input reasoning
can reduce their performance by up to 80 F1 points. Overall, the results
suggest that the in-distribution performance of smaller open-source models may
potentially rival GPT by incorporating appropriately structured derivation
dependencies during training, and highlight a shared weakness between BERT and
GPT involving a relative inability to decode indirect references to
mathematical entities. We release the full codebase, constructed datasets, and
fine-tuned models to encourage future progress in the field.
