  Commonly used speech corpora inadequately challenge academic and commercial
ASR systems. In particular, speech corpora lack metadata needed for detailed
analysis and WER measurement. In response, we present Earnings-21, a 39-hour
corpus of earnings calls containing entity-dense speech from nine different
financial sectors. This corpus is intended to benchmark ASR systems in the wild
with special attention towards named entity recognition. We benchmark four
commercial ASR models, two internal models built with open-source tools, and an
open-source LibriSpeech model and discuss their differences in performance on
Earnings-21. Using our recently released fstalign tool, we provide a candid
analysis of each model's recognition capabilities under different partitions.
Our analysis finds that ASR accuracy for certain NER categories is poor,
presenting a significant impediment to transcript comprehension and usage.
Earnings-21 bridges academic and commercial ASR system evaluation and enables
further research on entity modeling and WER on real world audio.
