Article: The Gospel According to Dixon #11: Heard Any Good Anecdotes Lately?

[Dixon]

We’ve all heard plenty of these tales: “I was troubled by (depression / indigestion / leprosy / the heartbreak of psoriasis / whatever) for years. I tried everything, but nothing helped. But then I met (Reverend / Swami / Doctor) Mountebanque, and just one session of (energy healing / crystals / potions / pills / quantum laser / shamanic techniques / powdered tiger penis / whatever) cured me! Plus, my sex life has improved, I’ve stopped snoring, and it leaves no bathtub ring!”

When such little stories, or “anecdotes”, are offered as evidence for something, that’s called anecdotal evidence (no surprise there). Often, most or all of the evidence proffered in support of a claim is anecdotal, and most of us, at one time or another, make decisions based on anecdotal evidence. Sometimes, it’s all we have to go by, but many people accept anecdotal evidence even when there’s better evidence to the contrary, apparently because they don’t realize just how undependable anecdotal evidence is. In fact, it’s so bad that it’s virtually no evidence at all.

Look at any of the gazillion examples of anecdotal evidence you can find on WaccoBB or almost anywhere. It’s much more emotionally vivid, more “human” feeling, than dry old scientific data. It’s natural for us to jump to the conclusion that such stories constitute good evidence. But unfortunately, that response is likely to be wrong, because it’s actually closed-minded! In other words, if we decide that the anecdote really means that Dr. Mountebanque’s treatment works, we’ve jumped to only one of several possible conclusions, without reasonably ruling the others out.

The next time you encounter seemingly persuasive anecdotes offered as evidence, I urge you to consider all of these possibilities:

1. The story is exactly true. The treatment works as described.

2. The story is entirely made up. It never really happened. (Yes, even seemingly cool and groovy “spiritual” folks are often not above lying to relieve suckers of their shekels.)

3. The person’s condition didn’t really improve, but they mistakenly experienced it as having improved. For instance, people may tell themselves, and really believe, that they’re smoking less or seeing better or being more productive even when it’s not true.

4. The person’s condition actually did improve after receiving the treatment, but the treatment itself had little or nothing to do with the improvement, which was caused by “nonspecific treatment effects” or factors entirely unrelated to the treatment.

These last possibilities deserve more discussion because most people don’t realize how easy it is for even the most intelligent, sane and honest of us to be fooled by these factors. For starters, let’s look at a few examples of “nonspecific treatment effects”:

• The more time/effort/money we invest in a treatment, the more likely we are to experience it as being helpful, even if it’s not really effective. This is called the “effort justification effect”.

• Especially with practitioners we like, we may tend to imagine or exaggerate benefits of the treatment so as not to make the “healer” feel ineffective and therefore sad or embarrassed. This is called the “please the therapist effect".

• Just the expectation that something will work can trigger real improvements. This is the famous “placebo effect”. Sometimes it can even be triggered in situations wherein one is reluctantly, skeptically trying something that’s purported to work. It can be pretty powerful, especially for subjective symptoms such as sleep or appetite problems, pain, depression, and anxiety, and has some startling features that illuminate our natural suggestibility. For instance, two sugar pills (i.e., physiologically inactive placebos) work better than one, and an injection of saline solution (another inactive placebo) works better than placebos in pill form!¹

• And, lest we forget, there’s the “confirmation bias”. In one of my earlier columns, I defined it thusly: “…our natural tendency to see things in ways that confirm what we want to believe...noticing, remembering and exaggerating confirming evidence; ignoring, forgetting, minimizing and making excuses for disconfirming evidence, and interpreting things as confirmation which really aren’t at all.”

• The very act of seeking treatment tends to create improvement (or sometimes the mistaken impression of improvement). Getting out of bed and going to see the “healer” implies some hope. Being with an empathic person perceived as having some expertise, who’s giving emotional support and maybe good advice, makes us feel at least a little better.

There’s also a relevant statistical effect called “regression toward the mean”: “[I]f a variable is extreme on its first measurement, it will tend to be closer to the average on a second measurement…”² Here’s how it relates to assessing things like the efficacy of treatment modalities: We usually seek treatment for our ills when we’re feeling lousy (i.e., toward the bad end of the spectrum of wellness). Statistically, the greatest probability is that shortly after we felt lousy enough to seek treatment, we’ll feel better (closer to our average level of wellness) regardless of whether we got effective treatment, useless treatment, or no treatment at all. But even if our feeling better has nothing to do with whatever useless treatment we received, nearly everyone tends to attribute the improvement to that last treatment. That’s the “post hoc, ergo propter hoc” fallacy (Latin for "after this, therefore because of this").

On top of all these, there are other confounding variables that may fool us into thinking a useless treatment has cured us. For instance, there could be a delayed effect from the second-to-last treatment we tried, which we mistakenly attribute to the more recent treatment. Or some treatment could have a negative effect on us, so that feeling better is a result of stopping it. Or any of a myriad of other factors, such as changes in diet, stress levels, or environmental insults to the immune system, or something else we’ll never identify (“coincidence”, “luck”) could account for our feeling better.

Not only that, but other psychological, cultural and social factors can reinforce our tendency to mistakenly attribute effectiveness to useless treatments. For instance, if we see using “alternative healers” as a way of resisting the evil depredations of patriarchy, the Establishment, or the greedy medical industry, we have strong ideological reinforcement, and probably lots of social reinforcement from our like-minded friends, for seeing such treatments as effective.³ In some social contexts, even questioning the efficacy of this or that treatment modality can be seen as a betrayal of subcultural norms, and the questioner regarded as a turncoat.

Some say: “What difference does it make whether it’s a placebo effect or an actual effect of the specific treatment itself? Either way, the person feels better.” But there’s a huge difference between concluding that our belief in a treatment can benefit us (which we already know) and concluding that Doctor Mountebanque’s Sacred Snake Oil Remedy can. These two conclusions have very different implications for our understanding of how the universe works and for our choices about what sorts of products, research, and enterprises to invest our time, money and energy in. One conclusion leads us ever further down the path of illusion, the other, down the path of truth.

Thus far I’ve focused on anecdotal evidence presented as support for the efficacy of treatments or products, but that’s not the only way it’s used. It’s also used to “prove” negative claims. Most of us have heard some version of the classic bigot’s anecdote: “I hate (blacks / whites / Mexicans / Jews / Muslims / druggies / men / women / homos / heteros / whoever) because some of them (robbed / raped / cheated / beat up / oppressed / discriminated against) me.”

Here again we see the implication that there’s a causative correlation between two things when there’s really no good reason to infer such a correlation from the anecdote(s). In the more favorable usage of anecdotal evidence, causative correlation is implied between a product or service and a positive outcome such as healing. In the unfavorable usage, the implied causation is between someone’s group status (ethnic, racial, gender, etc.) and some negative behavior. In both cases it’s the same logical fallacy. Other examples of merely anecdotal support for negative attributions could include some people’s claims that rock music or fantasy role-playing games like Dungeons and Dragons cause teenagers to commit suicide, that pot smoking causes stereotypical stupid behavior, that this or that substance is toxic, or that blondes are dumb.

Many are surprised to learn that the validity of anecdotal evidence increases very little, if at all, with an increase in the number of anecdotes. A thousand times zero is still just zero, and the plural of “anecdote” is not “evidence”. Citing hundreds or even thousands of instances wherein two things are associated does not establish a correlation, much less a causative one.

For example, some years ago I became aware of a large number of protesters who were angry that a presumed causative correlation between breast implants and several illnesses such as cancer was not being addressed by the government. They cited thousands of instances in which women who had implants also developed these illnesses, and felt that this constituted strong evidence for causation.

Here’s why they were wrong: Let’s say there are one million women who happen to be Presbyterian and who will also get cancer sometime in their lives. Just imagining them all presents a tragic, emotionally resonant picture of human suffering! Should we mount an angry demonstration demanding that the government ban Presbyterianism because it causes cancer? Of course not. Citing a million instances wherein Presbyterianism and cancer coincide in the same person doesn’t constitute evidence for a correlation between the two things any more than citing one such anecdote would! If our society includes, say, twenty million women who’ll have cancer during their lives, and five percent of the women in our society are Presbyterian, we can expect roughly one million cases of cancer in Presbyterian women due simply to the random distribution of the disease, with no correlation between cancer and Presbyterianism whatsoever!

So you see, anecdotes, whether one or a million, by themselves tell us nothing about any correlations, causative or otherwise. That’s why we have controlled scientific studies—to compare, e.g., women with breast implants to carefully selected control groups drawn from the general public, to see if there is a real correlation with cancer. Anecdotes have no proper place in this learning process, except to suggest what questions science should investigate next.

What should we do when encountering anecdotal evidence? Well, if it’s promoting bigoted generalities about groups of people, try to straighten out the nitwit who’s spouting it. If it’s testimonials about things we know really work, such as word-of-mouth from trusted friends about the skills of this auto mechanic or that carpenter, we may do well to believe it. But if someone is trying to convince you that some new pill, potion or technique works, ask how they’ve ruled out confounding factors like the placebo and effort justification effects. No good answer? Then ask for some logical reason to expect that the product or service will work, and run their explanation by someone you know who has expertise in the relevant science, or at least in critical thinking. If the claimant can’t give you more than rapturous testimonials, put your hand on your wallet and back away slowly.

NOTES

1. Here is an interesting and amusing short video on the subject.

2. Quoted from https://en.wikipedia.org/wiki/Regres...oward_the_mean

3. This argument is not to be taken as an endorsement of patriarchy, the Establishment, or the greedy medical industry!

About Dixon: I'm a hopeful monster, committed to laughter, love, and the Golden Rule. I see reason, applied with empathy, as the most important key to making a better world. I'm a lazy slob and a weirdo. I love cats, kids, quilts, fossils, tornadoes, comic books, unusual music, and too much else to mention. I’m a former conservative Christian, then New Ager, now a rationalist, skeptic and atheist. Lately I’m a Contributing Editor at the Omnificent English Dictionary In Limerick Form (That’s right!), and my humor is getting published in the Washington Post and Fantasy and Science Fiction. I’m job-hunting too, mostly in the Human Services realm. Passions: Too many -- Reading, writing, critical thinking, public speaking, human rights, sex and sensuality, arts and sciences, nature. Oh, and ladies, I’m single ;^D

[packardcarr]

A Threat Then A Question.

Threat: I'm gonna sue you for posting my picture without my written consent. Hmph!

Question: why do you make a point of telling the ladies you're single; don't you enjoy the freedom?

Packard

We’ve all heard plenty of these tales:

[Dixon]

A Threat Then A Question

Uh-oh!

Threat: I'm gonna sue you for posting my picture without my written consent. Hmph!

LOL! I doubt you're "Harry Pothead" (as opposed to a hairy pothead, of which there are plenty in this community), so you must be either the snake oil hawker or the blonde. If the latter--what are you doing Saturday night?

Question: why do you make a point of telling the ladies you're single; don't you enjoy the freedom?

Yes and no.

[Shandi]

Thanks for sharing your critical thinking.

I find that many MLM products are presented with success testimonials, in order to sell the product, which is usually priced higher than you would pay if purchased elsewhere.

We also know that there are many differences in our chemistry, which means some things work for some people, and don't work for others.

Hope and belief are big factors because they move our feelings to a higher level, way up from feeling despondent, depressed, hopeless. If anything or anyone can give us hope, we may be more receptive to the helper or potion that is presented. If we believe we feel better, that can make a huge difference.

I feel fortunate that one of my sons is a health practitioner in the field of NMT (neuromuscular therapy). He got into this field because of the helpful treatments he received. People rave about how he got them out of life-long pain. He is a very caring person, and just being around him helps people feel better. He doesn't promote himself; he gets clients from referrals.

He's always a good source of information for me if I'm having physical problems. I can't take advantage of his treatments since he lives in Orange County. But I've benefited from his continued learning.

Once when I was living with a "roommate from hell" it took 3 months, and $2,000 to get a judgement to get her out. When she was finally gone, I collapsed, and it was difficult for me to get out of bed. I felt as though I was dying. Friends said it was like "battle fatigue". My energy was drained, and I wasn't able to go to work.

I called my son, and he recommended that I start taking pharmaceutical grade DHEA. After the first 2 days of taking this, my energy started to return. I wasn't sure that it was making the difference, but it seemed to be working. I read about how stress can deplete the body. There seems to be evidence that as we age, our DHEA levels get reduced. I continue to take this, along with a few other supplements I buy on EBay or Amazon.

Another product he recommended for my painful arthritis, was a very expensive prescription only topical gel, called Voltaren. I had received painful cortisone shots that didn't last more than a few days, so I asked my Kaiser doc about Voltaren. He said he didn't know anyone who got relief from this, but gave me a prescription. It cost $85 a tube!!!

After rubbing a little of this non greasy, odorless, gel on my foot, I drove to the Healdsburg Salvation Army thrift stores with a friend. At this point, I had been unable to walk much without pain and swelling. When we arrived about 30 minutes later, and I stepped out of the car, I didn't feel the pain I usually feel. I couldn't believe it, and I told her "look, I can walk without pain!" I continued walking for about an hour before we left.

I did some research and found the product online from a Canadian pharmacy for about half of what I had paid, including shipping. I had to fax my prescription.

I learned that this product is sold over the counter in other countries for about $5, so I had a friend ship some to me from Australia. Leave it to the US to require an expensive prescription!

Since then I've found it on EBay without a prescription, from other countries. I've shared small amounts with about 20 friends, and each person has said it helped relieve their pain. It's a pain reliever and anti-inflammatory.

There is a pill form of this, but there's a caution for people with heart problems. I've gone online to read various testimonials from people who use the gel.

Because of this I'm able to walk more without pain. I was using it 4 times a day, but now I only use it when I know I need to do some walking to buy groceries or other errands. I've been able to avoid having the surgery my doctor recommended. That means a lot to me.

Keep up your good work with words!

We’ve all heard plenty of these tales: ...

[podfish]

too much math.. remember Barbie's wise observation.

seriously, I think that a big reason why anecdotal evidence is received the way it is, is because the math that underlays your explanation doesn't resonate emotionally with people at all. Correlation just feels way too much like causation - and it's a short-cut that peoples' minds are hardwired to rely on even when they understand its limitations.

[Dixon]

Thanks for sharing your experiences, Shandi. Of course, everything I said in the essay applies to the anecdotes you shared. Whether your and your friends' feeling better is due to nonspecific treatment effects, actual effects of the treatments themselves, or other factors, I'm glad you're feeling better! It would be interesting to see what, if anything, actual research studies may say about any of the remedies you mention, including neuromuscular therapy (whatever that is). I'm too ignorant on any of them to hazard an opinion, and haven't the time right now to research these treatments. All the best to you!

[Dixon]

too much math.. remember Barbie's wise observation.
seriously, I think that a big reason why anecdotal evidence is received the way it is, is because the math that underlays your explanation doesn't resonate emotionally with people at all. Correlation just feels way too much like causation - and it's a short-cut that peoples' minds are hardwired to rely on even when they understand its limitations.

Exactly! That's why I said, about anecdotal evidence, "It’s much more emotionally vivid, more 'human' feeling, than dry old scientific data."

[harpsterdave]

jealous?

[Dixon]

jealous?

Of what?

[Hotspring 44]

I am not very good at "math".
Jealous (?)... ...Who?... ...because?
harpsterdave; could you explain what you mean by that one word question?

jealous?