  Language models (LMs) are being scaled and becoming powerful. Improving their
efficiency is one of the core research topics in neural information processing
systems. Tay et al. (2022) provided a comprehensive overview of efficient
Transformers that have become an indispensable staple in the field of NLP.
However, in the section of "On Evaluation", they left an open question "which
fundamental efficient Transformer one should consider," answered by "still a
mystery" because "many research papers select their own benchmarks."
Unfortunately, there was not quantitative analysis about the performances of
Transformers on any benchmarks. Moreover, state space models (SSMs) have
demonstrated their abilities of modeling long-range sequences with
non-attention mechanisms, which were not discussed in the prior review. This
article makes a meta analysis on the results from a set of papers on efficient
Transformers as well as those on SSMs. It provides a quantitative review on LM
efficiency research and gives suggestions for future research.
