  A wide array of sequence models are built on a framework modeled after
Transformers, comprising alternating sequence mixer and channel mixer layers.
This paper studies a unifying matrix mixer view of sequence mixers that can be
conceptualized as a linear map on the input sequence. This framework
encompasses a broad range of well-known sequence models, including the
self-attention of Transformers as well as recent strong alternatives such as
structured state space models (SSMs), and allows understanding downstream
characteristics such as efficiency and expressivity through properties of their
structured matrix class. We identify a key axis of matrix parameterizations
termed sequence alignment, which increases the flexibility and performance of
matrix mixers, providing insights into the strong performance of Transformers
and recent SSMs such as Mamba. Furthermore, the matrix mixer framework offers a
systematic approach to developing sequence mixers with desired properties,
allowing us to develop several new sub-quadratic sequence models. In
particular, we propose a natural bidirectional extension of the Mamba model
(Hydra), parameterized as a quasiseparable matrix mixer, which demonstrates
superior performance over other sequence models including Transformers on
non-causal tasks. As a drop-in replacement for attention layers, Hydra
outperforms BERT by 0.8 points on the GLUE benchmark and ViT by 2% Top-1
accuracy on ImageNet.
