  Speech recognition applications cover a range of different audio and text
distributions, with different speaking styles, background noise, transcription
punctuation and character casing. However, many speech recognition systems
require dataset-specific tuning (audio filtering, punctuation removal and
normalisation of casing), therefore assuming a-priori knowledge of both the
audio and text distributions. This tuning requirement can lead to systems
failing to generalise to other datasets and domains. To promote the development
of multi-domain speech systems, we introduce the End-to-end Speech Benchmark
(ESB) for evaluating the performance of a single automatic speech recognition
(ASR) system across a broad set of speech datasets. Benchmarked systems must
use the same data pre- and post-processing algorithm across datasets - assuming
the audio and text data distributions are a-priori unknown. We compare a series
of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark,
demonstrating how a single speech system can be applied and evaluated on a wide
range of data distributions. We find E2E systems to be effective across
datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems
tuned to a specific dataset. Our analysis reveals that transcription artefacts,
such as punctuation and casing, pose difficulties for ASR systems and should be
included in evaluation. We believe E2E benchmarking over a range of datasets
promotes the research of multi-domain speech recognition systems. ESB is
available at https://huggingface.co/esb.
