  The transformer-based pre-trained language models have been tremendously
successful in most of the conventional NLP tasks. But they often struggle in
those tasks where numerical understanding is required. Some possible reasons
can be the tokenizers and pre-training objectives which are not specifically
designed to learn and preserve numeracy. Here we investigate the ability of
text-to-text transfer learning model (T5), which has outperformed its
predecessors in the conventional NLP tasks, to learn numeracy. We consider four
numeracy tasks: numeration, magnitude order prediction, finding minimum and
maximum in a series, and sorting. We find that, although T5 models perform
reasonably well in the interpolation setting, they struggle considerably in the
extrapolation setting across all four tasks.
