  Fact knowledge memorization is crucial for Large Language Models (LLM) to
generate factual and reliable responses. However, the behaviors of LLM fact
memorization remain under-explored. In this paper, we analyze the scaling laws
for LLM's fact knowledge and LLMs' behaviors of memorizing different types of
facts. We find that LLMs' fact knowledge capacity has a linear and negative
exponential law relationship with model size and training epochs, respectively.
Estimated by the built scaling law, memorizing the whole Wikidata's facts
requires training an LLM with 1000B non-embed parameters for 100 epochs,
suggesting that using LLMs to memorize all public facts is almost implausible
for a general pre-training setting. Meanwhile, we find that LLMs can generalize
on unseen fact knowledge and its scaling law is similar to general
pre-training. Additionally, we analyze the compatibility and preference of
LLMs' fact memorization. For compatibility, we find LLMs struggle with
memorizing redundant facts in a unified way. Only when correlated facts have
the same direction and structure, the LLM can compatibly memorize them. This
shows the inefficiency of LLM memorization for redundant facts. For preference,
the LLM pays more attention to memorizing more frequent and difficult facts,
and the subsequent facts can overwrite prior facts' memorization, which
significantly hinders low-frequency facts memorization. Our findings reveal the
capacity and characteristics of LLMs' fact knowledge learning, which provide
directions for LLMs' fact knowledge augmentation.
