  The amount of freely available systems for automatic speech recognition (ASR)
based on neural networks is growing steadily, with equally increasingly
reliable predictions. However, the evaluation of trained models is typically
exclusively based on statistical metrics such as WER or CER, which do not
provide any insight into the nature or impact of the errors produced when
predicting transcripts from speech input. This work presents a selection of ASR
model architectures that are pretrained on the German language and evaluates
them on a benchmark of diverse test datasets. It identifies cross-architectural
prediction errors, classifies those into categories and traces the sources of
errors per category back into training data as well as other sources. Finally,
it discusses solutions in order to create qualitatively better training
datasets and more robust ASR systems.
