  Large Language Models (LLMs), trained on massive corpora with billions of
parameters, show unprecedented performance in various fields. Though surprised
by their excellent performances, researchers also noticed some special
behaviors of those LLMs. One of those behaviors is memorization, in which LLMs
can generate the same content used to train them. Though previous research has
discussed memorization, the memorization of LLMs still lacks explanation,
especially the cause of memorization and the dynamics of generating them. In
this research, we comprehensively discussed memorization from various
perspectives and extended the discussion scope to not only just the memorized
content but also less and unmemorized content. Through various studies, we
found that: (1) Through experiments, we revealed the relation of memorization
between model size, continuation size, and context size. Further, we showed how
unmemorized sentences transition to memorized sentences. (2) Through embedding
analysis, we showed the distribution and decoding dynamics across model size in
embedding space for sentences with different memorization scores. The n-gram
statistics analysis presents d (3) An analysis over n-gram and entropy decoding
dynamics discovered a boundary effect when the model starts to generate
memorized sentences or unmemorized sentences. (4)We trained a Transformer model
to predict the memorization of different models, showing that it is possible to
predict memorizations by context.
