  Recent deep learning approaches have achieved impressive performance on
speech enhancement and separation tasks. However, these approaches have not
been investigated for separating mixtures of arbitrary sounds of different
types, a task we refer to as universal sound separation, and it is unknown how
performance on speech tasks carries over to non-speech tasks. To study this
question, we develop a dataset of mixtures containing arbitrary sounds, and use
it to investigate the space of mask-based separation architectures, varying
both the overall network architecture and the framewise analysis-synthesis
basis for signal transformations. These network architectures include
convolutional long short-term memory networks and time-dilated convolution
stacks inspired by the recent success of time-domain enhancement networks like
ConvTasNet. For the latter architecture, we also propose novel modifications
that further improve separation performance. In terms of the framewise
analysis-synthesis basis, we explore both a short-time Fourier transform (STFT)
and a learnable basis, as used in ConvTasNet. For both of these bases, we also
examine the effect of window size. In particular, for STFTs, we find that
longer windows (25-50 ms) work best for speech/non-speech separation, while
shorter windows (2.5 ms) work best for arbitrary sounds. For learnable bases,
shorter windows (2.5 ms) work best on all tasks. Surprisingly, for universal
sound separation, STFTs outperform learnable bases. Our best methods produce an
improvement in scale-invariant signal-to-distortion ratio of over 13 dB for
speech/non-speech separation and close to 10 dB for universal sound separation.
