  A Virtual Patient (VP) is a powerful tool for training medical students to
take patient histories, where responding to a diverse set of spoken questions
is essential to simulate natural conversations with a student. The performance
of such a Spoken Language Understanding system (SLU) can be adversely affected
by both the presence of Automatic Speech Recognition (ASR) errors in the test
data and a high degree of class imbalance in the SLU training data. While these
two issues have been addressed separately in prior work, we develop a novel
two-step training methodology that tackles both these issues effectively in a
single dialog agent. As it is difficult to collect spoken data from users
without a functioning SLU system, our method does not rely on spoken data for
training, rather we use an ASR error predictor to "speechify" the text data.
Our method shows significant improvements over strong baselines on the VP
intent classification task at various word error rate settings.
