  Speech enhancement and separation are two fundamental tasks for robust speech
processing. Speech enhancement suppresses background noise while speech
separation extracts target speech from interfering speakers. Despite a great
number of supervised learning-based enhancement and separation methods having
been proposed and achieving good performance, studies on applying
self-supervised learning (SSL) to enhancement and separation are limited. In
this paper, we evaluate 13 SSL upstream methods on speech enhancement and
separation downstream tasks. Our experimental results on Voicebank-DEMAND and
Libri2Mix show that some SSL representations consistently outperform baseline
features including the short-time Fourier transform (STFT) magnitude and log
Mel filterbank (FBANK). Furthermore, we analyze the factors that make existing
SSL frameworks difficult to apply to speech enhancement and separation and
discuss the representation properties desired for both tasks. Our study is
included as the official speech enhancement and separation downstreams for
SUPERB.
