  Large language models (LLMs) trained on web-scale datasets raise substantial
concerns regarding permissible data usage. One major question is whether these
models "memorize" all their training data or they integrate many data sources
in some way more akin to how a human would learn and synthesize information.
The answer hinges, to a large degree, on how we define memorization. In this
work, we propose the Adversarial Compression Ratio (ACR) as a metric for
assessing memorization in LLMs. A given string from the training data is
considered memorized if it can be elicited by a prompt (much) shorter than the
string itself -- in other words, if these strings can be "compressed" with the
model by computing adversarial prompts of fewer tokens. The ACR overcomes the
limitations of existing notions of memorization by (i) offering an adversarial
view of measuring memorization, especially for monitoring unlearning and
compliance; and (ii) allowing for the flexibility to measure memorization for
arbitrary strings at a reasonably low compute. Our definition serves as a
practical tool for determining when model owners may be violating terms around
data usage, providing a potential legal tool and a critical lens through which
to address such scenarios.
