  We explore the critical data size in language models, a threshold that marks
a fundamental shift from quick memorization to slow generalization. We
formalize the phase transition under the grokking configuration into the Data
Efficiency Hypothesis and identify data insufficiency, sufficiency, and surplus
regimes in language models training dynamics. We develop a grokking
configuration to reproduce grokking on simplistic language models stably by
rescaling initialization and weight decay. We show that generalization occurs
only when language models reach a critical size. We analyze grokking across
sample-wise and model-wise, verifying the proposed data efficiency hypothesis.
Our experiments reveal smoother phase transitions occurring at the critical
dataset size for language datasets. As the model size increases, this critical
point also becomes larger, indicating that larger models require more data. Our
results deepen the understanding of language model training, offering a novel
perspective on the role of data in the learning mechanism of language models.
