  The Conformer has become the most popular encoder model for automatic speech
recognition (ASR). It adds convolution modules to a transformer to learn both
local and global dependencies. In this work we describe a faster, more
memory-efficient, and better-performing transformer, called Zipformer. Modeling
changes include: 1) a U-Net-like encoder structure where middle stacks operate
at lower frame rates; 2) reorganized block structure with more modules, within
which we re-use attention weights for efficiency; 3) a modified form of
LayerNorm called BiasNorm allows us to retain some length information; 4) new
activation functions SwooshR and SwooshL work better than Swish. We also
propose a new optimizer, called ScaledAdam, which scales the update by each
tensor's current scale to keep the relative change about the same, and also
explictly learns the parameter scale. It achieves faster convergence and better
performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and
WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer
over other state-of-the-art ASR models. Our code is publicly available at
https://github.com/k2-fsa/icefall.
