  Natural language numbers are an example of compositional structures, where
larger numbers are composed of operations on smaller numbers. Given that
compositional reasoning is a key to natural language understanding, we propose
novel multilingual probing tasks tested on DistilBERT, XLM, and BERT to
investigate for evidence of compositional reasoning over numerical data in
various natural language number systems. By using both grammaticality judgment
and value comparison classification tasks in English, Japanese, Danish, and
French, we find evidence that the information encoded in these pretrained
models' embeddings is sufficient for grammaticality judgments but generally not
for value comparisons. We analyze possible reasons for this and discuss how our
tasks could be extended in further studies.
