The Truth Wears Off

from delancyplace.com:

"On September 18, 2007, a few dozen neuroscientists, psychiatrists, and drug-company
executives gathered in a hotel conference room in Brussels to hear some startling
news. It had to do with a class of drugs known as atypical or second-generation
antipsychotics, which came on the market in the early nineties. The drugs, sold
under brand names such as Abilify, Seroquel, and Zyprexa, had been tested on schizophrenics
in several large clinical trials, all of which had demonstrated a dramatic decrease
in the subjects' psychiatric symptoms. As a result, second-generation antipsychotics
had become one of the fastest-growing and most profitable pharmaceutical classes.
By 2001, Eli Lilly's Zyprexa was generating more revenue than Prozac. It remains
the company's top-selling drug.
"But the data presented at the Brussels meeting made it clear that something strange
was happening: the therapeutic power of the drugs appeared to be steadily waning.
A recent study showed an effect that was less than half of that documented in the
first trials, in the early nineteen-nineties. Many researchers began to argue that
the expensive pharmaceuticals weren't any better than first-generation antipsychotics,
which have been in use since the fifties. 'In fact, sometimes they now look even
worse,' John Davis, a professor of psychiatry at the University of Illinois at
Chicago, told me.
"Before the effectiveness of a drug can be confirmed, it must be tested and tested
again. Different scientists in different labs need to repeat the protocols and publish
their results. The test of replicability, as it's known, is the foundation of modern
research. Replicability is how the community enforces itself. It's a safeguard for
the creep of subjectivity. Most of the time, scientists know what results they want,
and that can influence the results they get. The premise of replicability is that
the scientific community can correct for these flaws.
"But now all sorts of well-established, multiply confirmed findings have started
to look increasingly uncertain. It's as if our facts were losing their truth: claims
that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn't
yet have an official name, but it's occurring across a wide range of fields, from
psychology to ecology. In the field of medicine, the phenomenon seems extremely
widespread, affecting not only antipsychotics but also therapies ranging from cardiac
stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating
that the efficacy of antidepressants has gone down as much as threefold in recent
decades.
For many scientists, the effect is especially troubling because of what it exposes
about the scientific process. If replication is what separates the rigor of science
from the squishiness of pseudoscience, where do we put all these rigorously validated
findings that can no longer be proved? Which results should we believe? Francis
Bacon, the early-modern philosopher and pioneer of the scientific method, once declared
that experiments were essential, because they allowed us to 'put nature to the question.'
But it appears that nature often gives us different answers. ...
"[Joseph Banks Rhine, a psychologist at Duke, came to call this trend toward a reduction
in the strength of proof for a theory he had developed in the early nineteen-thirties]
the 'decline effect.'
"According to John Ioannidis, an epidemiologist at Stanford University, the main
problem is that too many researchers engage in what he calls 'significance chasing,'
or finding ways to interpret the data so that it passes the statistical test of
significance - the ninety-five-per-cent boundary invented by Ronald Fisher. 'The
scientists are so eager to pass this magical test that they start playing around
with the numbers, trying to find anything that seems worthy,' Ioannidis says. In
recent years, Ioannidis has become increasingly blunt about the pervasiveness of
the problem. One of his most cited papers has a deliberately provocative title:
'Why Most Published Research Findings Are False.'
"The problem of selective reporting is rooted in a fundamental cognitive flaw, which
is that we like proving ourselves right and hate being wrong. 'It feels good to
validate a hypothesis,' Ioannidis said. 'It feels even better when you've got a
financial interest in the idea or your career depends upon it. And that's why, even
after a claim has been systematically disproven' - he cites, for instance, the early
work on hormone replacement therapy, or claims involving various vitamins - 'you
still see some stubborn researchers citing the first few studies that show a strong
effect. They really want to believe that it's true.' ...
"The disturbing implication of a study [conducted in the late nineteen-nineties
by John Crabbe, a neuroscientist at the Oregon Health and Science University] is
that a lot of extraordinary scientific data are nothing but noise. The problem,
of course, is that ... dramatic findings are ... the most likely to get published
in prestigious journals, since the data are both statistically significant and entirely
unexpected. Grants get written, follow-up studies are conducted. The end result
is a scientific accident that can take years to unravel.
"This suggests that the decline effect is actually a decline of illusion. While
Karl Popper imagined falsification occurring with a single, definitive experiment
- Galileo refuted Aristotelian mechanics in an afternoon - the process turns out
to be much messier than that. Many scientific theories continue to be considered
true even after failing numerous experimental tests. Verbal overshadowing might
exhibit the decline effect, but it remains extensively relied upon within the field.
The same holds for any number of phenomena, from the disappearing benefits of second-generation
antipsychotics to the weak coupling ratio exhibited by decaying neutrons, which
appears to have fallen by more than ten standard deviations between 1969 and 2001....
"Such anomalies demonstrate the slipperiness of empiricism. Although many scientific
ideas generate conflicting results and suffer from falling effect sizes, they continue
to get cited in the textbooks and drive standard medical practice. Why? Because
these ideas seem true. Because they make sense. Because we can't bear to let them
go. And this is why the decline effect is so troubling. Not because it reveals the
human fallibility of science, in which data are tweaked and beliefs shape perceptions.
(Such shortcomings aren't surprising, at least for scientists.) And not because
it reveals that many of our most exciting theories are fleeting fads and will soon
be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is
troubling because it reminds us how difficult it is to prove anything. We like to
pretend that our experiments define the truth for us. But that's often not the case.
Just because an idea is true doesn't mean it can be proved. And just because an
idea can be proved doesn't mean it's true. When the experiments are done, we still
have to choose what to believe."
Author: Jonah Lehrer
Title: "The Truth Wears Off"
Publisher: The New Yorker
Date: December 13, 2010
Pages: 52-57