  Single channel target speaker separation (TSS) aims at extracting a speaker's
voice from a mixture of multiple talkers given an enrollment utterance of that
speaker. A typical deep learning TSS framework consists of an upstream model
that obtains enrollment speaker embeddings and a downstream model that performs
the separation conditioned on the embeddings. In this paper, we look into
several important but overlooked aspects of the enrollment embeddings,
including the suitability of the widely used speaker identification embeddings,
the introduction of the log-mel filterbank and self-supervised embeddings, and
the embeddings' cross-dataset generalization capability. Our results show that
the speaker identification embeddings could lose relevant information due to a
sub-optimal metric, training objective, or common pre-processing. In contrast,
both the filterbank and the self-supervised embeddings preserve the integrity
of the speaker information, but the former consistently outperforms the latter
in a cross-dataset evaluation. The competitive separation and generalization
performance of the previously overlooked filterbank embedding is consistent
across our study, which calls for future research on better upstream features.
