  Deep learning models such as the Transformer are often constructed by
heuristics and experience. To provide a complementary foundation, in this work
we study the following problem: Is it possible to find an energy function
underlying the Transformer model, such that descent steps along this energy
correspond with the Transformer forward pass? By finding such a function, we
can view Transformers as the unfolding of an interpretable optimization process
across iterations. This unfolding perspective has been frequently adopted in
the past to elucidate more straightforward deep models such as MLPs and CNNs;
however, it has thus far remained elusive obtaining a similar equivalence for
more complex models with self-attention mechanisms like the Transformer. To
this end, we first outline several major obstacles before providing companion
techniques to at least partially address them, demonstrating for the first time
a close association between energy function minimization and deep layers with
self-attention. This interpretation contributes to our intuition and
understanding of Transformers, while potentially laying the ground-work for new
model designs.
