  Speech data from different domains has distinct acoustic and linguistic
characteristics. It is common to train a single multidomain model such as a
Conformer transducer for speech recognition on a mixture of data from all
domains. However, changing data in one domain or adding a new domain would
require the multidomain model to be retrained. To this end, we propose a
framework called modular domain adaptation (MDA) that enables a single model to
process multidomain data while keeping all parameters domain-specific, i.e.,
each parameter is only trained by data from one domain. On a streaming
Conformer transducer trained only on video caption data, experimental results
show that an MDA-based model can reach similar performance as the multidomain
model on other domains such as voice search and dictation by adding per-domain
adapters and per-domain feed-forward networks in the Conformer encoder.
