  Large language models (LLMs) have achieved remarkable proficiency on solving
diverse problems. However, their generalization ability is not always
satisfying and the generalization problem is common for generative transformer
models in general. Researchers take basic mathematical tasks like n-digit
addition or multiplication as important perspectives for investigating their
generalization behaviors. It is observed that when training models on n-digit
operations (e.g., additions) in which both input operands are n-digit in
length, models generalize successfully on unseen n-digit inputs
(in-distribution (ID) generalization), but fail miserably on longer, unseen
cases (out-of-distribution (OOD) generalization). We bring this unexplained
performance drop into attention and ask whether there is systematic OOD
generalization. Towards understanding LLMs, we train various smaller language
models which may share the same underlying mechanism. We discover that the
strong ID generalization stems from structured representations, while behind
the unsatisfying OOD performance, the models still exhibit clear learned
algebraic structures. Specifically, these models map unseen OOD inputs to
outputs with learned equivalence relations in the ID domain, which we call the
equivalence generalization. These findings deepen our knowledge regarding the
generalizability of generative models including LLMs, and provide insights into
potential avenues for improvement.
