  Standard accuracy metrics have shown that Math Word Problem (MWP) solvers
have achieved high performance on benchmark datasets. However, the extent to
which existing MWP solvers truly understand language and its relation with
numbers is still unclear. In this paper, we generate adversarial attacks to
evaluate the robustness of state-of-the-art MWP solvers. We propose two methods
Question Reordering and Sentence Paraphrasing to generate adversarial attacks.
We conduct experiments across three neural MWP solvers over two benchmark
datasets. On average, our attack method is able to reduce the accuracy of MWP
solvers by over 40 percentage points on these datasets. Our results demonstrate
that existing MWP solvers are sensitive to linguistic variations in the problem
text. We verify the validity and quality of generated adversarial examples
through human evaluation.
