  Inspired by the recent progress in self-supervised learning for computer
vision, in this paper we introduce DeLoRes, a new general-purpose audio
representation learning approach. Our main objective is to make our network
learn representations in a resource-constrained setting (both data and
compute), that can generalize well across a diverse set of downstream tasks.
Inspired from the Barlow Twins objective function, we propose to learn
embeddings that are invariant to distortions of an input audio sample, while
making sure that they contain non-redundant information about the sample. To
achieve this, we measure the cross-correlation matrix between the outputs of
two identical networks fed with distorted versions of an audio segment sampled
from an audio file and make it as close to the identity matrix as possible. We
use a combination of a small subset of the large-scale AudioSet dataset and
FSD50K for self-supervised learning and are able to learn with less than half
the parameters compared to state-of-the-art algorithms. For evaluation, we
transfer these learned representations to 9 downstream classification tasks,
including speech, music, and animal sounds, and show competitive results under
different evaluation setups. In addition to being simple and intuitive, our
pre-training algorithm is amenable to compute through its inherent nature of
construction and does not require careful implementation details to avoid
trivial or degenerate solutions. Furthermore, we conduct ablation studies on
our results and make all our code and pre-trained models publicly available
https://github.com/Speech-Lab-IITM/DeLoRes.
