It is vital, first, to distinguish between two types of proposition that anecdotes can be claimed to be evidence for: causal and observational propositions. A causal claim is of the form “P caused X” (or “P, Q and R caused X”) and an observational proposition is “P happened” (or “P occurred, then Q happen and after that R”). This distinction is important for a simple reason: an anecdote can in isolation (almost) never establish the truth or falsehood of a causal claim, it can only be evidence for observational propositions. Why this is the case will be clearer if we think through some concrete examples.
Consider the claim “Mary cheated on John with Bob”. This kind of proposition is pretty straightforward. I can convince myself it is true simply by determining whether Mary and John are in an exclusive relationship, and, if I see Mary make out with Bob, I can reasonably conclude the proposition is true. Now, obviously, there are a bunch of ways I could get it wrong: maybe the woman involved wasn’t Mary, maybe it was simply a friendly hello kiss, or maybe Mary had broken up with John (so it’s making out, but not cheating). It’s clear, though, that if I’m just a little careful, there are a wide variety of circumstances in which I could be very confident Mary did in fact cheat based on my personal experiences. There are a couple of extra complications when John has to decide whether to believe my testimony – maybe I’m lying, for example – but, again, these are not difficult to understand even if they’re difficult to deal with in practice. In other words, we here have a clear case of an anecdote – both in the sense of personal experience and testimony – that can be a good reason to accept the truth of some proposition. Notice two things, though: the claim is not a causal one, and the plausibility (or prior probability) of it being true is pretty high since we know people do in fact cheat on each other regularly. I’ll explain what the latter means in a bit, but let’s move on to anecdotes as evidence for causal claims.
So consider the causal proposition “I took medicine X, I got better, therefore I got better because of medicine X”. This claim is much more complicated than the one about Mary, and you can’t determine whether it’s true simply by looking or making a few observations. Why? Because causality is counterfactual: to say A caused B, is to say B would not have occurred if A had not occurred. (There is a long-standing and complicated philosophical debate about causality. Take it from me: steer clear and stick to the counterfactual view). So, on our example, when you claim “medicine X caused my recovery” you’re committed to the counterfactual proposition that you wouldn’t have gotten better had you not taken medicine X. And you simply cannot, even in principle, know this. For one thing, you have an immune system (gasp!) which may have fought off your infection irrespective of whether you took the medicine. Alternatively, you could have ingested some other substance - maybe you took medicine Y as well, maybe you ate something therapeutic - that took care of the infection. In other words, there are a whole bunch of things - i.e. confounds - that could have caused your recovery, and a single observation – the source of an anecdote – simply cannot distinguish between them. In general, the only reliable way to establish the truth or falsity of causal claims is to do controlled experiments. Let’s look at this a bit more closely.
Assume we are trying to understand the causal relationship between four dichotomous and independent variables (A-D) and a particular dichotomous outcome (O). Assume also that these four variables exhaust the universe of all variables even conceivably related to the outcome. Our aim is to make either inclusion inferences (i.e. conclude the relevant variable has a causal relationship to the outcome) or exclusion inferences (i.e. that the variable does not have a causal relationship to the outcome). Given these assumptions, inclusion inferences are valid only when, from:
A1 B1 C1 D1 => O1
A2 B1 C1 D1 => O2
it is concluded that A1 caused O1. The inference is valid because all the variables except one (A) was controlled – held constant – and given the outcome changed, it follows that A is causally related to O. Exclusion inferences are valid only when, from:
A1 B1 C1 D1 => O1
A2 B1 C1 D1 => O1
it is concluded A1 is not causally related to O1. Again, the validity of the argument is assured because all the variables save one (A) was controlled. Given that A varies while O does not, it follows that A is not causally related to O. Notice that in both cases we are comparing one outcome with a counterfactual. The process of engineering such comparisons is called experimentation and it is at the very heart of the scientific method. (These, by the way, are versions of Mill's Methods).
If we want to determine whether medicine X can cure some disease, we cannot rely on anecdotes because they don’t allow for counterfactual comparison. Assume, for example, that A1 is taking medicine X, and A2 is not taking it; that B1 is having an immune system, and B2 not; that C1 is taking medicine Y, and C2 not taking it; that D1 is being overweight, D2 is not being overweight; and, finally, that O1 is getting better and O2 is not getting better. When we have a single observation - we know that Thaba over there took medicine X, has a healthy immune system, that he isn't taking medicine Y, that he's rather overweight and that he got better after a few days - all we have is:
A1 B1 C1 D2 => O1
We have no proper counterfactual: we have not controlled for variables B, C, or D so we can't make any logical inferences about variable A. (Making this inference is the post hoc fallacy). At best, we can say that taking the medicine is possibly related to getting better, but then the same goes for B, C and D. Note also that adding more anecdotes does not resolve our problem: in the real world there are many more than just four variables so things are much more complicated, experiments involving humans are always possibly confounded by the placebo effect, and, importantly, the variables may interact in complex ways. In the oft repeated phrase, the plural of anecdote is anecdotes, not data. Thousands of anecdotes are no more convincing that a single anecdote. As a result, then, anecdotes cannot in general (that is, barring extreme exceptions) establish the truth or falsity of causal propositions.
As I showed above anecdotes can reasonably be taken as convincing evidence for observational claims. But that does not mean we should believe every anecdote (concerning observational propositions). Bob Carrol of Skepdic (an excellent resource worth referring to often, by the way) nicely enumerates the possible problems:
Anecdotes are unreliable for various reasons. Stories are prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on. Most stories get distorted in the telling and the retelling. Events get exaggerated. Time sequences get confused. Details get muddled. Memories are imperfect and selective; they are often filled in after the fact. People misinterpret their experiences. Experiences are conditioned by biases, memories, and beliefs, so people's perceptions might not be accurate. Most people aren't expecting to be deceived, so they may not be aware of deceptions that others might engage in. Some people make up stories. Some stories are delusions. Sometimes events are inappropriately deemed psychic simply because they seem improbable when they might not be that improbable after all. In short, anecdotes are inherently problematic and are usually impossible to test for accuracy.In other words, while anecdotes can be good evidence for believing observational propositions - "x happened" - for the reasons listed above, we certainly can't accept all anecdotes uncritically. So what to do? Life is impossible if we dismiss all anecdotes, but we'll be led astray if we accept all anecdotes. The solution is skepticism: that is, being open minded but then filtering beliefs through a bullshit detector. Doing this is simple in principle, but incredibly difficult in practice, so some examples are in order. (Recommended books: Thinking About Thinking by Anthony Flew, The Demon-Haunted World by Carl Sagan [my review], Truth by Simon Blackburn [my review], and Mistakes Were Made (But Not by Me) by Travis & Aronson [my review]).
|Megan Fox: not in Jeff's league.|
A somewhat more enlightening example is alien abduction. People from all over the world claim to have been abducted by extraterrestrials and then molested, lectured on the necessity of world peace, and so on. So, step one: how plausible are these claims? Given what we know about physics and human psychology, not very. First of all, we currently have no evidence that life - let alone intelligent life - exists anywhere else in the universe. (My gut tells me alien life is abundant, but as Carl Sagan pointed out, we shouldn't think with our guts). Secondly, if intelligent life does exist, the aliens will in all likelihood be tens of light-years or more distant, and, since we have no reason to think faster than light travel is practicable, there is no known way for aliens to get to earth in a reasonable period of time. Step two: what about the evidence? Again we have anecdotes: tales from people who claim to have been abducted. Significantly for this example, there is a highly plausible alternative explanation that undermines the evidentiary status of the accounts, namely, hypnogogia and hympnopompia. Briefly, these are vivid hallucinations that occur as you're falling asleep or waking up that are accompanied by sleep paralysis. Typically, a person wakes up terrified and unable to move, senses (an often malevolent) 'presence' in the room, and may also experience a variety of visual, auditory and proprioceptionary hallucinations. Tellingly, this well-studied phenomenon closely mirrors accounts of alien abduction, which often feature extreme fear, a 'presence' in the room, and being unable to move. Since these experiences are accompanied by visual hallucinations and alien visitation is a common trope in popular culture, the other reported experiences are easily accounted for. Importantly, also, hypnogogia and hypnopompia are common (much more common than claims of alien abduction). Extraordinary claims require extraordinary evidence. Since claims of abduction are extraordinary, anecdotes of being abducted are far from extraordinary evidence, so, until much more evidence is provided, it is reasonable to withhold assent.
So... what exactly is the evidentiary status of anecdotes? In summary: (1) anecdotes on their own can never establish the truth or falsity of causal propositions. (2) While anecdotes can be evidence for observational propositions, the plausibility of the claim must be taken into account. To be believed, highly implausible claims require much, much more than mere anecdotes.