  For humans, language production and comprehension is sensitive to the
hierarchical structure of sentences. In natural language processing, past work
has questioned how effectively neural sequence models like transformers capture
this hierarchical structure when generalizing to structurally novel inputs. We
show that transformer language models can learn to generalize hierarchically
after training for extremely long periods -- far beyond the point when
in-domain accuracy has saturated. We call this phenomenon \emph{structural
grokking}. On multiple datasets, structural grokking exhibits inverted U-shaped
scaling in model depth: intermediate-depth models generalize better than both
very deep and very shallow transformers. When analyzing the relationship
between model-internal properties and grokking, we find that optimal depth for
grokking can be identified using the tree-structuredness metric of
\citet{murty2023projections}. Overall, our work provides strong evidence that,
with extended training, vanilla transformers discover and use hierarchical
structure.
