  State-of-the-art pretrained language models tend to perform below their
capabilities when applied out-of-the-box on tasks that require understanding
and working with numbers. Recent work suggests two main reasons for this: (1)
popular tokenisation algorithms have limited expressiveness for numbers, and
(2) common pretraining objectives do not target numeracy. Approaches that
address these shortcomings usually require architectural changes or pretraining
from scratch. In this paper, we propose a new extended pretraining approach
called Arithmetic-Based Pretraining that jointly addresses both in one extended
pretraining step without requiring architectural changes or pretraining from
scratch. Arithmetic-Based Pretraining combines contrastive learning to improve
the number representation, and a novel extended pretraining objective called
Inferable Number Prediction Task to improve numeracy. Our experiments show the
effectiveness of Arithmetic-Based Pretraining in three different tasks that
require improved numeracy, i.e., reading comprehension in the DROP dataset,
inference-on-tables in the InfoTabs dataset, and table-to-text generation in
the WikiBio and SciGen datasets.
