  Automated speaker identification (SID) is a crucial step for the
personalization of a wide range of speech-enabled services. Typical SID systems
use a symmetric enrollment-verification framework with a single model to derive
embeddings both offline for voice profiles extracted from enrollment
utterances, and online from runtime utterances. Due to the distinct
circumstances of enrollment and runtime, such as different computation and
latency constraints, several applications would benefit from an asymmetric
enrollment-verification framework that uses different models for enrollment and
runtime embedding generation. To support this asymmetric SID where each of the
two models can be updated independently, we propose using a lightweight neural
network to map the embeddings from the two independent models to a shared
speaker embedding space. Our results show that this approach significantly
outperforms cosine scoring in a shared speaker logit space for models that were
trained with a contrastive loss on large datasets with many speaker identities.
This proposed Neural Embedding Speaker Space Alignment (NESSA) combined with an
asymmetric update of only one of the models delivers at least 60% of the
performance gain achieved by updating both models in the standard symmetric SID
approach.
