  Automatic speech recognition (ASR) models trained on large amounts of audio
data are now widely used to convert speech to written text in a variety of
applications from video captioning to automated assistants used in healthcare
and other domains. As such, it is important that ASR models and their use is
fair and equitable. Prior work examining the performance of commercial ASR
systems on the Corpus of Regional African American Language (CORAAL)
demonstrated significantly worse ASR performance on African American English
(AAE). The current study seeks to understand the factors underlying this
disparity by examining the performance of the current state-of-the-art neural
network based ASR system (Whisper, OpenAI) on the CORAAL dataset. Two key
findings have been identified as a result of the current study. The first
confirms prior findings of significant dialectal variation even across
neighboring communities, and worse ASR performance on AAE that can be improved
to some extent with fine-tuning of ASR models. The second is a novel finding
not discussed in prior work on CORAAL: differences in audio recording practices
within the dataset have a significant impact on ASR accuracy resulting in a
``confounding by provenance'' effect in which both language use and recording
quality differ by study location. These findings highlight the need for further
systematic investigation to disentangle the effects of recording quality and
inherent linguistic diversity when examining the fairness and bias present in
neural ASR models, as any bias in ASR accuracy may have negative downstream
effects on disparities in various domains of life in which ASR technology is
used.
