  We conduct a large scale empirical investigation of contextualized number
prediction in running text. Specifically, we consider two tasks: (1)masked
number prediction-predicting a missing numerical value within a sentence, and
(2)numerical anomaly detection-detecting an errorful numeric value within a
sentence. We experiment with novel combinations of contextual encoders and
output distributions over the real number line. Specifically, we introduce a
suite of output distribution parameterizations that incorporate latent
variables to add expressivity and better fit the natural distribution of
numeric values in running text, and combine them with both recurrent and
transformer-based encoder architectures. We evaluate these models on two
numeric datasets in the financial and scientific domain. Our findings show that
output distributions that incorporate discrete latent variables and allow for
multiple modes outperform simple flow-based counterparts on all datasets,
yielding more accurate numerical prediction and anomaly detection. We also show
that our models effectively utilize textual con-text and benefit from
general-purpose unsupervised pretraining.
