  Transformers have seen an unprecedented rise in Natural Language Processing
and Computer Vision tasks. However, in audio tasks, they are either infeasible
to train due to extremely large sequence length of audio waveforms or incur a
performance penalty when trained on Fourier-based features. In this work, we
introduce an architecture, Audiomer, where we combine 1D Residual Networks with
Performer Attention to achieve state-of-the-art performance in keyword spotting
with raw audio waveforms, outperforming all previous methods while being
computationally cheaper and parameter-efficient. Additionally, our model has
practical advantages for speech processing, such as inference on arbitrarily
long audio clips owing to the absence of positional encoding. The code is
available at https://github.com/The-Learning-Machines/Audiomer-PyTorch.
