  Word Error Rate (WER) is the primary metric used to assess automatic speech
recognition (ASR) model quality. It has been shown that ASR models tend to have
much higher WER on speakers with speech impairments than typical English
speakers. It is hard to determine if models can be be useful at such high error
rates. This study investigates the use of BERTScore, an evaluation metric for
text generation, to provide a more informative measure of ASR model quality and
usefulness. Both BERTScore and WER were compared to prediction errors manually
annotated by Speech Language Pathologists for error type and assessment.
BERTScore was found to be more correlated with human assessment of error type
and assessment. BERTScore was specifically more robust to orthographic changes
(contraction and normalization errors) where meaning was preserved.
Furthermore, BERTScore was a better fit of error assessment than WER, as
measured using an ordinal logistic regression and the Akaike's Information
Criterion (AIC). Overall, our findings suggest that BERTScore can complement
WER when assessing ASR model performance from a practical perspective,
especially for accessibility applications where models are useful even at lower
accuracy than for typical speech.
