  Previous works have evaluated memorization by comparing model outputs with
training corpora, examining how factors such as data duplication, model size,
and prompt length influence memorization. However, analyzing these extensive
training corpora is highly time-consuming. To address this challenge, this
paper proposes an innovative approach named ROME that bypasses direct
processing of the training data. Specifically, we select datasets categorized
into three distinct types -- context-independent, conventional, and factual --
and redefine memorization as the ability to produce correct answers under these
conditions. Our analysis then focuses on disparities between memorized and
non-memorized samples by examining the logits and representations of generated
texts. Experimental findings reveal that longer words are less likely to be
memorized, higher confidence correlates with greater memorization, and
representations of the same concepts are more similar across different
contexts. Our code and data will be publicly available when the paper is
accepted.
