  Transformers have revolutionized deep learning across various tasks,
including audio representation learning, due to their powerful modeling
capabilities. However, they often suffer from quadratic complexity in both GPU
memory usage and computational inference time, affecting their efficiency.
Recently, state space models (SSMs) like Mamba have emerged as a promising
alternative, offering a more efficient approach by avoiding these complexities.
Given these advantages, we explore the potential of SSM-based models in audio
tasks. In this paper, we introduce Self-Supervised Audio Mamba (SSAMBA), the
first self-supervised, attention-free, and SSM-based model for audio
representation learning. SSAMBA leverages the bidirectional Mamba to capture
complex audio patterns effectively. We incorporate a self-supervised
pretraining framework that optimizes both discriminative and generative
objectives, enabling the model to learn robust audio representations from
large-scale, unlabeled datasets. We evaluated SSAMBA on various tasks such as
audio classification, keyword spotting, and speaker identification. Our results
demonstrate that SSAMBA outperforms the Self-Supervised Audio Spectrogram
Transformer (SSAST) in most tasks. Notably, SSAMBA is approximately 92.7%
faster in batch inference speed and 95.4% more memory-efficient than SSAST for
the tiny model size with an input token size of 22k. These efficiency gains,
combined with superior performance, underscore the effectiveness of SSAMBA's
architectural innovation, making it a compelling choice for a wide range of
audio processing applications.
