  Prior work has attempted to understand the internal structures and
functionalities of Transformer-based encoder-decoder architectures on the level
of multi-head attention and feed-forward sublayers. Interpretations have
focused on the encoder and decoder, along with the combinatorial possibilities
of the self-attention, cross-attention, and feed-forward sublayers. However,
without examining the low-level structures, one gains limited understanding of
the motivation behind sublayer reordering. Could we dive into the sublayer
abstraction and permute layer weight matrices to improve the quality of
translation? We propose AEIUOrder to greedily reorder layer weight matrices in
the encoder by their well-trainedness, as measured by Heavy-Tailed
Self-Regularization (HT-SR) metrics, and order the decoder matrices
correspondingly. Our results suggest that greedily reordering layer weight
matrices to maximize Total well-trainedness facilitates the model to learn
representations and generate translations more effectively.
