  Recent years have seen progress beyond domain-specific sound separation for
speech or music towards universal sound separation for arbitrary sounds. Prior
work on universal sound separation has investigated separating a target sound
out of an audio mixture given a text query. Such text-queried sound separation
systems provide a natural and scalable interface for specifying arbitrary
target sounds. However, supervised text-queried sound separation systems
require costly labeled audio-text pairs for training. Moreover, the audio
provided in existing datasets is often recorded in a controlled environment,
causing a considerable generalization gap to noisy audio in the wild. In this
work, we aim to approach text-queried universal sound separation by using only
unlabeled data. We propose to leverage the visual modality as a bridge to learn
the desired audio-textual correspondence. The proposed CLIPSep model first
encodes the input query into a query vector using the contrastive
language-image pretraining (CLIP) model, and the query vector is then used to
condition an audio separation model to separate out the target sound. While the
model is trained on image-audio pairs extracted from unlabeled videos, at test
time we can instead query the model with text inputs in a zero-shot setting,
thanks to the joint language-image embedding learned by the CLIP model.
Further, videos in the wild often contain off-screen sounds and background
noise that may hinder the model from learning the desired audio-textual
correspondence. To address this problem, we further propose an approach called
noise invariant training for training a query-based sound separation model on
noisy data. Experimental results show that the proposed models successfully
learn text-queried universal sound separation using only noisy unlabeled
videos, even achieving competitive performance against a supervised model in
some settings.
