  Despite recent advancements in audio-text modeling, audio-text contrastive
models still lag behind their image-text counterparts in scale and performance.
We propose a method to improve both the scale and the training of audio-text
contrastive models. Specifically, we craft a large-scale audio-text dataset
containing 13,000 hours of text-labeled audio, using pretrained language models
to process noisy text descriptions and automatic captioning to obtain text
descriptions for unlabeled audio samples. We first train on audio-only data
with a masked autoencoder (MAE) objective, which allows us to benefit from the
scalability of unlabeled audio datasets. We then, initializing our audio
encoder from the MAE model, train a contrastive model with an auxiliary
captioning objective. Our final model, which we name Cacophony, achieves
state-of-the-art performance on audio-text retrieval tasks, and exhibits
competitive results on the HEAR benchmark and other downstream tasks such as
zero-shot classification.
