  The Transformer architecture has become prominent in developing large causal
language models. However, mechanisms to explain its capabilities are not well
understood. Focused on the training process, here we establish a meta-learning
view of the Transformer architecture when trained for the causal language
modeling task, by explicating an inner optimization process within the
Transformer. Further, within the inner optimization, we discover and
theoretically analyze a special characteristic of the norms of learned token
representations within Transformer-based causal language models. Our analysis
is supported by experiments in various settings.
