  Physical measurements constitute a large portion of numbers in academic
papers, engineering reports, and web tables. Current benchmarks fall short of
properly evaluating numeracy of pretrained language models on measurements,
hindering research on developing new methods and applying them to numerical
tasks. To that end, we introduce a novel task, Masked Measurement Prediction
(MMP), where a model learns to reconstruct a number together with its
associated unit given masked text. MMP is useful for both training new
numerically informed models as well as evaluating numeracy of existing systems.
In order to address this task, we introduce a new Generative Masked Measurement
(GeMM) model that jointly learns to predict numbers along with their units. We
perform fine-grained analyses comparing our model with various ablations and
baselines. We use linear probing of traditional pretrained transformer models
(RoBERTa) to show that they significantly underperform jointly trained
number-unit models, highlighting the difficulty of this new task and the
benefits of our proposed pretraining approach. We hope this framework
accelerates the progress towards building more robust numerical reasoning
systems in the future.
