  Transformers have become one of the most important architectural innovations
in deep learning and have enabled many breakthroughs over the past few years.
Here we propose a simple network architecture, gMLP, based on MLPs with gating,
and show that it can perform as well as Transformers in key language and vision
applications. Our comparisons show that self-attention is not critical for
Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model
achieves parity with Transformers on pretraining perplexity and is better on
some downstream NLP tasks. On finetuning tasks where gMLP performs worse,
making the gMLP model substantially larger can close the gap with Transformers.
In general, our experiments show that gMLP can scale as well as Transformers
over increased data and compute.
