  Linearization of attention using various kernel approximation and kernel
learning techniques has shown promise. Past methods use a subset of
combinations of component functions and weight matrices within the random
features paradigm. We identify the need for a systematic comparison of
different combinations of weight matrix and component functions for attention
learning in Transformer. In this work, we introduce Spectraformer, a unified
framework for approximating and learning the kernel function in linearized
attention of the Transformer. We experiment with broad classes of component
functions and weight matrices for three textual tasks in the LRA benchmark. Our
experimentation with multiple combinations of component functions and weight
matrices leads us to a novel combination with 23.4% faster training time and
25.2% lower memory consumption over the previous SOTA random feature
Transformer, while maintaining the performance, as compared to the Original
Transformer. Our code is available at:
https://github.com/dukeraphaelng/spectraformer .
