  Permutation Invariant Training (PIT) has long been a stepping stone method
for training speech separation model in handling the label ambiguity problem.
With PIT selecting the minimum cost label assignments dynamically, very few
studies considered the separation problem to be optimizing both the model
parameters and the label assignments, but focused on searching for good model
architecture and parameters. In this paper, we investigate instead for a given
model architecture the various flexible label assignment strategies for
training the model, rather than directly using PIT. Surprisingly, we discover a
significant performance boost compared to PIT is possible if the model is
trained with fixed label assignments and a good set of labels is chosen. With
fixed label training cascaded between two sections of PIT, we achieved the
state-of-the-art performance on WSJ0-2mix without changing the model
architecture at all.
