  Modules in all existing speech separation networks can be categorized into
single-input-multi-output (SIMO) modules and single-input-single-output (SISO)
modules. SIMO modules generate more outputs than input, and SISO modules keep
the numbers of input and output the same. While the majority of separation
models only contain SIMO architectures, it has also been shown that certain
two-stage separation systems integrated with a post-enhancement SISO module can
improve the separation quality. Why performance improvements can be achieved by
incorporating the SISO modules? Are SIMO modules always necessary? In this
paper, we empirically examine those questions by designing models with varying
configurations in the SIMO and SISO modules. We show that comparing with the
standard SIMO-only design, a mixed SIMO-SISO design with a same model size is
able to improve the separation performance especially under low-overlap
conditions. We further validate the necessity of SIMO modules and show that
SISO-only models are still able to perform separation without sacrificing the
performance. The observations allow us to rethink the model design paradigm and
present different views on how the separation is performed.
