  We introduce EdgeFormer -- a parameter-efficient Transformer for on-device
seq2seq generation under the strict computation and memory constraints.
Compared with the previous parameter-efficient Transformers, EdgeFormer applies
two novel principles for cost-effective parameterization, allowing it to
perform better given the same parameter budget; moreover, EdgeFormer is further
enhanced by layer adaptation innovation that is proposed for improving the
network with shared layers.
  Extensive experiments show EdgeFormer can effectively outperform previous
parameter-efficient Transformer baselines and achieve competitive results under
both the computation and memory constraints. Given the promising results, we
release EdgeLM -- the pretrained version of EdgeFormer, which is the first
publicly available pretrained on-device seq2seq model that can be easily
fine-tuned for seq2seq tasks with strong results, facilitating on-device
seq2seq generation in practice.
