  Knowledge distillation has widely been used for model compression and domain
adaptation for speech applications. In the presence of multiple teachers,
knowledge can easily be transferred to the student by averaging the models
output. However, previous research shows that the student do not adapt well
with such combination. This paper propose to use an elitist sampling strategy
at the output of ensemble teacher models to select the best-decoded utterance
generated by completely out-of-domain teacher models for generalizing unseen
domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the
student is adapted for the Switchboard data. The results show that with the
selection strategy based on the individual models posteriors the student model
achieves a better WER compared to all the teachers and baselines with a minimum
absolute improvement of about 8.4 percent. Furthermore, an insights on the
model adaptation with out-of-domain data has also been studied via correlation
analysis.
