Any time there is discussion of alternative medicine, of treatments that haven’t passed the test of rigorous scientific investigation (or even been tested at all), there are miracle stories. Testimonials, anecdotes, whatever you call them, they are tales of amazing recoveries from illness when everything tried had failed and all hope was lost. The comments on this blog contain many such stories for almost all of the products and methods I have questioned.
And such stories provide emotionally compelling evidence for the effectiveness of these interventions. If an owner has tried a hundred treatments for their pet, or been told there is no hope and nothing can save the animal, and then they try one more thing and their companion recovers, surely this means that final treatment must have worked? And yet, with all these testaments to the power of these alternative methods, I still have doubts. Let’s take a look at why that is.
There are a number of arguments against believing that testimonials prove anything about the effects of a medical treatment. Most of these, of course, will have no effect on the individual experiencing the miracle. It is in the nature of human beings to find their own experiences, and stories from others, deeply compelling, and sadly facts and logical arguments have little impact on that fundamentally emotional and irrational sense of certainty. Still, making the effort to understand the limitations of anecdotes has led us to this point in time, a point in which our medicine is more effective by far than any time in human history. So there is clearly value in understanding the problem of miracle stories even if many people will still find them compelling.
Things Aren’t Always What They Seem
The first classic argument is simply that things aren’t always what they seem. There are many reasons why the fact that a patient gets better after receiving a particular medical treatment doesn’t actually tell us the treatment made the patient better. The notion that if an action preceding an outcome means the action must have caused that outcome is known as the post hoc ergo propter hoc fallacy. In medicine, there is a long list of reasons why this kind of reasoning about causation isn’t trustworthy:
- Self-limiting Disease:
Most medical problems ultimately go away by themselves. Certainly minor ones nearly always do, and even major, life-threatening diseases like cancer resolve spontaneously despite our belief that they can’t or shouldn’t. It is natural to assume that whatever we did last, right before the patient got better, is the reason they got better. Natural, but not reliably true.
- Regression to the Mean:
Chronic diseases, especially mysterious ones we don’t fully understand, tend to come and go. In addition to spontaneously going away altogether, which they can even after many years, they commonly tend to get better for a while then worse for a while and back and forth in an unpredictable cycle which we are always trying to understand and predict. Some stories of miraculous results from medical treatments are simply cases in which the patient was having a particularly bad run of symptoms and then felt better after getting treated.
If we systematically followed up all these stories, we would find a fair number of people whose symptoms returned despite continuing the supposedly effective treatment. Just like we find that almost any fad diet works for some people, and yet by a year most of them have gained back the weight they lost initially. Companies and practitioners selling alternative therapies, however, don’t make a point of systematically following up on all their satisfied customers’ testimonials to see which eventually got sick again. I wonder why that is?
- Placebo Effect:
Everyone hates this one because it means you believe something which isn’t actually true. You use a treatment and feel better, but if it’s just a placebo and your disease isn’t actually any different, that means you must be crazy or stupid, right? No! Claiming that a perceived change is only the result of a placebo effect isn’t an insult to the patient. It is simply a fact that all human beings see want we want to see a fair portion of the time, and that sometimes we interpret what we see to mean something that it doesn’t really mean.
In the case of medical therapies, almost anything I give a client, they will perceive some improvement right after simply because we’ve done something and we know we are supposed to be looking hard for any sign of an effect, especially since we really want there to be one. Again, systematically following up all these early responses often reveals no meaningful change, but if you ask at the right time, the answer will almost always be positive and suitable for use as a testimonial.
Sometimes, the reason a disease went away after an unconventional treatment (or a conventional one, for that matter) was used is because the disease was never there to begin with. Doctors aren’t perfect, and they sometimes diagnose diseases the patient doesn’t have. Any certainly people who aren’t trained medical professionals but who diagnose themselves or their pets are quite likely wrong a lot of the time. It’s easy to cure a disease that a patient didn’t have, so one of the first questions to ask about any miracle story is what hard, objective, tangible evidence is there to support the diagnosis? If your dog had cancer and it was cured by some kind of pseudoscientific nonsense like homeopathy, there had better be extensive imaging and before and after biopsies with long-term follow up before you start claiming homeopathy actually works and all the laws of physics and chemistry are mistakes.
You Can’t Prove It Isn’t a Miracle!
Another problem with miracle stories as proof that specific medical treatments actually work is that such stories can’t ever be proven wrong. And if something can’t be proven wrong no matter what, then you have to believe each and every one equally, which as we will see is probably the biggest problem with anecdotes.
Of course, the main reason you can’t disprove a miracle tale is psychological; cognitive dissonance is such a powerful and ubiquitous feature of the human brain, that it is almost impossible to convince someone they are wrong about a strongly emotional personal experience. Just as it is a great relief and comfort to know that you saved your beloved pet by using an unconventional therapy your vet said wouldn’t work, it is equally terrifying to believe that actually the therapy had nothing to do with your pet getting better and that not only did you waste your money and put your pet at unnecessary risk, but the world is ultimately unpredictable and uncontrollable and “As flies to wanton boys, are we to the gods. They kill us for their sport.” (King Lear) People cling to their miracles largely because these stories give them hope that bad things can be avoided or controlled if we just find the right things to do or believe.
Another reason miracle stories are unfalsifiable, which makes them largely useless as a form of evidence in science, is that they consist of very limited and selective information. By definition, anecdotes presented to show a treatment works contain only information that supports the effectiveness of the treatment. It wouldn’t be a miracle if at the end of the story the disease killed the patient. But does that mean there aren’t stories about the same treatment in which there was no benefit? Or even great harm? No, of course not. Testimonials are a form of “evidence” that is positive by definition, and that necessarily excludes negative evidence no matter how much of it might actually exist.
Of course, for a true believer, it would make no difference even if one could produce a story of failure for every single story of success. Cognitive dissonance would simply move the believer to say, “Well, maybe it doesn’t work for everybody, but it sure worked for me!” But for anyone trying to make an objective decision about how likely a particular therapy is to be effective, it is worth noting that testimonials and other anecdotes leave out all the stories that show the therapy failing, so they are an inherently biased kind of evidence.
Though it doesn’t always show up on the typical lists of reasons why we shouldn’t trust miracle stories, to me the most damning argument against them is simply that they can be found supporting every medical treatment ever invented. On this blog alone, you can find a testimonial in the comments for nearly every product or therapy I have questioned, despite the fact that most of them have entirely different theories and methods by which they are supposed to work. And if you look at alternative medicine in general, there is not a single practice that you can’t find lots of people with powerful, emotional stories showing positive results.
But let’s go beyond the alternative therapies popular in the US today. Sure, there are plenty of testimonials for, say, Double Helix Water. How about for other kinds of magic water?
So does every kind of water cure disease? Maybe the problem isn’t that testimonials aren’t unreliable. Maybe we’ve just stumbled across the fact that water itself is medicine? Ah, but I wonder then if we can find the same testimonials for every other therapy ever invented? Let’s see.