  When only limited target domain data is available, domain adaptation could be
used to promote performance of deep neural network (DNN) acoustic model by
leveraging well-trained source model and target domain data. However, suffering
from domain mismatch and data sparsity, domain adaptation is very challenging.
This paper proposes a novel adaptation method for DNN acoustic model using
class similarity. Since the output distribution of DNN model contains the
knowledge of similarity among classes, which is applicable to both source and
target domain, it could be transferred from source to target model for the
performance improvement. In our approach, we first compute the frame level
posterior probabilities of source samples using source model. Then, for each
class, probabilities of this class are used to compute a mean vector, which we
refer to as mean soft labels. During adaptation, these mean soft labels are
used in a regularization term to train the target model. Experiments showed
that our approach outperforms fine-tuning using one-hot labels on both accent
and noise adaptation task, especially when source and target domain are highly
mismatched.
