  The performance of automatic speech recognition (ASR) systems can be
significantly compromised by previously unseen conditions, which is typically
due to a mismatch between training and testing distributions. In this paper, we
address robustness by studying domain invariant features, such that domain
information becomes transparent to ASR systems, resolving the mismatch problem.
Specifically, we investigate a recent model, called the Factorized Hierarchical
Variational Autoencoder (FHVAE). FHVAEs learn to factorize sequence-level and
segment-level attributes into different latent variables without supervision.
We argue that the set of latent variables that contain segment-level
information is our desired domain invariant feature for ASR. Experiments are
conducted on Aurora-4 and CHiME-4, which demonstrate 41% and 27% absolute word
error rate reductions respectively on mismatched domains.
