  Modern Automatic Speech Recognition (ASR) systems often use a portfolio of
domain-specific models in order to get high accuracy for distinct user
utterance types across different devices. In this paper, we propose an
innovative approach that integrates the different per-domain per-device models
into a unified model, using a combination of domain embedding, domain experts,
mixture of experts and adversarial training. We run careful ablation studies to
show the benefit of each of these innovations in contributing to the accuracy
of the overall unified model. Experiments show that our proposed unified
modeling approach actually outperforms the carefully tuned per-domain models,
giving relative gains of up to 10% over a baseline model with negligible
increase in the number of parameters.
