  Measuring automatic speech recognition (ASR) system quality is critical for
creating user-satisfying voice-driven applications. Word Error Rate (WER) has
been traditionally used to evaluate ASR system quality; however, it sometimes
correlates poorly with user perception/judgement of transcription quality. This
is because WER weighs every word equally and does not consider semantic
correctness which has a higher impact on user perception. In this work, we
propose evaluating ASR output hypotheses quality with SemDist that can measure
semantic correctness by using the distance between the semantic vectors of the
reference and hypothesis extracted from a pre-trained language model. Our
experimental results of 71K and 36K user annotated ASR output quality show that
SemDist achieves higher correlation with user perception than WER. We also show
that SemDist has higher correlation with downstream Natural Language
Understanding (NLU) tasks than WER.
