  In this work, we analyze various scaling limits of the training dynamics of
transformer models in the feature learning regime. We identify the set of
parameterizations that admit well-defined infinite width and depth limits,
allowing the attention layers to update throughout training--a relevant notion
of feature learning in these models. We then use tools from dynamical mean
field theory (DMFT) to analyze various infinite limits (infinite key/query
dimension, infinite heads, and infinite depth) which have different statistical
descriptions depending on which infinite limit is taken and how attention
layers are scaled. We provide numerical evidence of convergence to the limits
and discuss how the parameterization qualitatively influences learned features.
