  The "Switchboard benchmark" is a very well-known test set in automatic speech
recognition (ASR) research, establishing record-setting performance for systems
that claim human-level transcription accuracy. This work highlights
lesser-known practical considerations of this evaluation, demonstrating major
improvements in word error rate (WER) by correcting the reference
transcriptions and deviating from the official scoring methodology. In this
more detailed and reproducible scheme, even commercial ASR systems can score
below 5% WER and the established record for a research system is lowered to
2.3%. An alternative metric of transcript precision is proposed, which does not
penalize deletions and appears to be more discriminating for human vs. machine
performance. While commercial ASR systems are still below this threshold, a
research system is shown to clearly surpass the accuracy of commercial human
speech recognition. This work also explores using standardized scoring tools to
compute oracle WER by selecting the best among a list of alternatives. A phrase
alternatives representation is compared to utterance-level N-best lists and
word-level data structures; using dense lattices and adding out-of-vocabulary
words, this achieves an oracle WER of 0.18%.
