  Numerical Question Answering is the task of answering questions that require
numerical capabilities. Previous works introduce general adversarial attacks to
Numerical Question Answering, while not systematically exploring numerical
capabilities specific to the topic. In this paper, we propose to conduct
numerical capability diagnosis on a series of Numerical Question Answering
systems and datasets. A series of numerical capabilities are highlighted, and
corresponding dataset perturbations are designed. Empirical results indicate
that existing systems are severely challenged by these perturbations. E.g.,
Graph2Tree experienced a 53.83% absolute accuracy drop against the ``Extra''
perturbation on ASDiv-a, and BART experienced 13.80% accuracy drop against the
``Language'' perturbation on the numerical subset of DROP. As a counteracting
approach, we also investigate the effectiveness of applying perturbations as
data augmentation to relieve systems' lack of robust numerical capabilities.
With experiment analysis and empirical studies, it is demonstrated that
Numerical Question Answering with robust numerical capabilities is still to a
large extent an open question. We discuss future directions of Numerical
Question Answering and summarize guidelines on future dataset collection and
system design.
