  A long-standing question in automatic speech recognition research is how to
attribute errors to the ability of a model to model the acoustics, versus its
ability to leverage higher-order context (lexicon, morphology, syntax,
semantics). We validate a novel approach which models error rates as a function
of relative textual predictability, and yields a single number, $k$, which
measures the effect of textual predictability on the recognizer. We use this
method to demonstrate that a Wav2Vec 2.0-based model makes greater stronger use
of textual context than a hybrid ASR model, in spite of not using an explicit
language model, and also use it to shed light on recent results demonstrating
poor performance of standard ASR systems on African-American English. We
demonstrate that these mostly represent failures of acoustic--phonetic
modelling. We show how this approach can be used straightforwardly in
diagnosing and improving ASR.
