  Since the success of a time-domain speech separation, further improvements
have been made by expanding the length and channel of a feature sequence to
increase the amount of computation. When temporally expanded to a long
sequence, the feature is segmented into chunks as a dual-path model in most
studies of speech separation. In particular, it is common for the process of
separating features corresponding to each speaker to be located in the final
stage of the network. However, it is more advantageous and intuitive to
proactively expand the feature sequence to include the number of speakers as an
extra dimension. In this paper, we present an asymmetric strategy in which the
encoder and decoder are partitioned to perform distinct processing in
separation tasks. The encoder analyzes features, and the output of the encoder
is split into the number of speakers to be separated. The separated sequences
are then reconstructed by the weight-shared decoder, as Siamese network, in
addition to cross-speaker processing. By using the Siamese network in the
decoder, without using speaker information, the network directly learns to
discriminate the features using a separation objective. With a common split
layer, intermediate encoder features for skip connections are also split for
the reconstruction decoder based on the U-Net structure. In addition, instead
of segmenting the feature into chunks as dual-path, we design global and local
Transformer blocks to directly process long sequences. The experimental results
demonstrated that this separation-and-reconstruction framework is effective and
that the combination of proposed global and local Transformer can sufficiently
replace the role of inter- and intra-chunk processing in dual-path structure.
Finally, the presented model including both of these achieved state-of-the-art
performance with less computation than before in various benchmark datasets.
