  Various natural language processing (NLP) tasks necessitate models that are
efficient and small based on their ultimate application at the edge or in other
resource-constrained environments. While prior research has reduced the size of
these models, increasing computational efficiency without considerable
performance impacts remains difficult, especially for autoregressive tasks.
This paper proposes modular linearized attention (MLA), which combines multiple
efficient attention mechanisms, including cosFormer, to maximize inference
quality while achieving notable speedups. We validate this approach on several
autoregressive NLP tasks, including speech-to-text neural machine translation
(S2T NMT), speech-to-text simultaneous translation (SimulST), and
autoregressive text-to-spectrogram, noting efficiency gains on TTS and
competitive performance for NMT and SimulST during training and inference.
