The Blind (Mis-) Leading the Blind
The latest in this long sad
history is a double-blind test that, the authors conclude, demonstrates that
44.1kHz/16-bit digital audio is indistinguishable from high-resolution
digital. Note the word "indistinguishable." The authors aren't saying that
high-res digital might sound a little different from Red Book CD but is no
better. Or that high-res digital is only slightly better and not worth the
additional cost. Rather, they reach the startling conclusion that CD-quality
audio sounds exactly the same as 96kHz/24-bit PCM and DSD, the encoding scheme
used in SACD. That is, under double-blind test conditions, 60 expert listeners
over 554 trials couldn't hear any differences between CD, SACD, and 96/24. The
study was published in the September, 2007 Journal of the Audio Engineering
Such tests are an indictment of blind listening in general
because of the patently absurd conclusions to which they lead. A notable
example is the blind listening test conducted by Stereo Review that concluded
that a pair of Mark Levinson monoblocks, an output-transformerless tubed
amplifier, and a $220 Pioneer receiver were sonically identical. ("Do All
Amplifiers Sound the Same?" published in the January, 1987 issue.)
Most such tests, including this new CD vs. high-res
comparison, are performed not by disinterested experimenters but by partisan
hacks bent on discrediting audiophiles. But blind listening tests lead to the
wrong conclusions even when the experimenters' motives are pure. A good
example is the listening tests conducted by Swedish Radio to decide whether
one of the low-bit-rate codecs under consideration by the European Broadcast
Union was good enough to replace FM broadcasting in Europe.
Swedish Radio developed an elaborate listening methodology
called "double-blind, triple-stimulus, hidden-reference." A "subject"
(listener) would hear three "objects" (musical presentations); presentation A
was always the unprocessed signal, with the listener required to identify if
presentation B or C had been processed through the codec.
The test involved 60 "expert" listeners spanning 20,000
evaluations over a period of two years. Swedish Radio announced in 1991 that
it had narrowed the field to two codecs, and that "both codecs have now
reached a level of performance where they fulfill the EBU requirements for a
distribution codec." In other words, the codecs were good enough to replace
analog FM broadcasts in Europe.
After announcing its decision, Swedish Radio sent a tape of
music processed by the selected codec to the late Bart Locanthi, an expert in
digital audio and chairman of an ad hoc committee formed to independently
evaluate low-bit-rate codecs. Using the same non-blind observational listening
techniques that audiophiles routinely use to evaluate sound quality, Locanthi
instantly identified an artifact of the codec. After Locanthi informed Swedish
Radio of the artifact (an idle tone at 1.5kHz), listeners at Swedish Radio
also instantly heard the distortion.
How is it possible that a single listener, using non-blind
observational listening techniques, was able to discover — in less than ten
minutes — a distortion that escaped the scrutiny of 60 expert listeners,
20,000 trials conducted over a two-year period, and elaborate "double-blind,
triple-stimulus, hidden-reference" methodology? The answer is that blind
listening tests fundamentally distort the listening process.
As exemplified by yet another reader letter published in
this issue, many people naively assume that blind listening tests are somehow
more rigorous and honest than the "single-presentation" observational
listening protocols practiced in product reviewing — that the undeniable value
of blind studies of new drugs, for example, automatically confers utility on
blind listening tests.
I've thought quite a bit about this subject and written what
I hope is a reasoned analysis of why blind listening tests are flawed. This
analysis is part of a larger statement on critical listening in general, which
I presented in a paper to the Audio Engineering Society entitled "The Role of
Critical Listening in Evaluating Audio Equipment Quality." A revised and
updated version of the paper is now available on AVguide.com.
I invite readers to comment on the paper, and discuss blind
listening tests, on the Forum of AVguide.com. We've created a new section
called "Evaluation, Measurement, Testing, and Perception: Discussion of how to
evaluate products, how to report on that evaluation, and link that evaluation
to real experience/value." I look forward to reading your ideas.