  For a language model (LM) to faithfully model human language, it must
compress vast, potentially infinite information into relatively few dimensions.
We propose analyzing compression in (pre-trained) LMs from two points of view:
geometric and information-theoretic. We demonstrate that the two views are
highly correlated, such that the intrinsic geometric dimension of linguistic
data predicts their coding length under the LM. We then show that, in turn,
high compression of a linguistic dataset predicts rapid adaptation to that
dataset, confirming that being able to compress linguistic information is an
important part of successful LM performance. As a practical byproduct of our
analysis, we evaluate a battery of intrinsic dimension estimators for the first
time on linguistic data, showing that only some encapsulate the relationship
between information-theoretic compression, geometric compression, and
ease-of-adaptation.
