  In this work we provide new insights into the transformer architecture, and
in particular, its best-known variant, BERT. First, we propose a method to
measure the degree of non-linearity of different elements of transformers.
Next, we focus our investigation on the feed-forward networks (FFN) inside
transformers, which contain 2/3 of the model parameters and have so far not
received much attention. We find that FFNs are an inefficient yet important
architectural element and that they cannot simply be replaced by attention
blocks without a degradation in performance. Moreover, we study the
interactions between layers in BERT and show that, while the layers exhibit
some hierarchical structure, they extract features in a fuzzy manner. Our
results suggest that BERT has an inductive bias towards layer commutativity,
which we find is mainly due to the skip connections. This provides a
justification for the strong performance of recurrent and weight-shared
transformer models.
