Auditing Studies of Anti-Depressants
posted by Frank Pasquale
Marcia Angell has kicked off another set of controversies for the pharmaceutical sector in two recent review essays in the New York Review of Books. She favorably reviews meta-research that calls into question the effectiveness of many antidepressant drugs:
Kirsch and his colleagues used the Freedom of Information Act to obtain FDA reviews of all placebo-controlled clinical trials, whether positive or negative, submitted for the initial approval of the six most widely used antidepressant drugs approved between 1987 and 1999—Prozac, Paxil, Zoloft, Celexa, Serzone, and Effexor. . . .Altogether, there were forty-two trials of the six drugs. Most of them were negative. Overall, placebos were 82 percent as effective as the drugs, as measured by the Hamilton Depression Scale (HAM-D), a widely used score of symptoms of depression. The average difference between drug and placebo was only 1.8 points on the HAM-D, a difference that, while statistically significant, was clinically meaningless. The results were much the same for all six drugs: they were all equally unimpressive. Yet because the positive studies were extensively publicized, while the negative ones were hidden, the public and the medical profession came to believe that these drugs were highly effective antidepressants.
Angell discusses other research that indicates that placebos can often be nearly as effective as drugs for conditions like depression. Psychiatrist Peter Kramer, a long-time advocate of anti-depressant therapy, responded to her last Sunday. He admits that “placebo responses . . . have been steadily on the rise” in FDA data; “in some studies, 40 percent of subjects not receiving medication get better.” But he believes that is only because the studies focus on the mildly depressed:
The problem is so big that entrepreneurs have founded businesses promising to identify genuinely ill research subjects. The companies use video links to screen patients at central locations where (contrary to the practice at centers where trials are run) reviewers have no incentives for enrolling subjects. In early comparisons, off-site raters rejected about 40 percent of subjects who had been accepted locally — on the ground that those subjects did not have severe enough symptoms to qualify for treatment. If this result is typical, many subjects labeled mildly depressed in the F.D.A. data don’t have depression and might well respond to placebos as readily as to antidepressants.
Yves Smith finds Kramer’s response unconvincing:
The research is clear: the efficacy of antidepressants is (contrary to what [Kramer's] article suggests) lower than most drugs (70% is a typical efficacy rate; for antidepressants, it’s about 50%. The placebo rate is 20% to 30% for antidepressants). And since most antidepressants produce side effects, patients in trials can often guess successfully as to whether they are getting real drugs. If a placebo is chosen that produces a symptom, say dry mouth, the efficacy of antidepressants v. placebos is almost indistinguishable. The argument made in [Kramer's] article to try to deal with this inconvenient fact, that many of the people chosen for clinical trials really weren’t depressed (thus contending that the placebo effect was simply bad sampling) is utter[ly wrong]. You’d see the mildly/short-term depressed people getting both placebos and real drugs. You would therefore expect to see the efficacy rate of both the placebo and the real drug boosted by the inclusion of people who just happened to get better anyhow.
Felix Salmon also challenges Kramer’s logic:
[Kramer's view is that] lots of people were diagnosed with depression and put onto a trial of antidepressant drugs, even when they were perfectly healthy. Which sounds very much like the kind of thing that Angell is complaining about: the way in which, for instance, the number of children so disabled by mental disorders that they qualify for Supplemental Security Income (SSI) or Social Security Disability Insurance (SSDI) was 35 times higher in 2007 than it was in 1987. And it’s getting worse: the editors of DSM-V, to be published in 2013, have written that “in primary care settings, approximately 30 percent to 50 percent of patients have prominent mental health symptoms or identifiable mental disorders, which have significant adverse consequences if left untreated.”
Those who would defend psychopharmacology, then, seem to want to have their cake and eat it: on the one hand it seems that serious mental health disorders have reached pandemic proportions, but on the other hand we’re told that a lot of people diagnosed with those disorders never really had them in the first place.
That is a very challenging point for the industry to consider as it responds to concerns like Angell’s. The diagnosis of mental illness will always have ineradicably economic dimensions and politically contestable aims. But doctors and researchers should insulate professional expertise and the interpretation of maladies as much as possible from inappropriate pressures.
How can they maintain that kind of independent clinical judgment? I think one key is to assure that data from all trials is open to all researchers. Consider, for instance, these findings from a NEJM study on “selective publication:”
We obtained reviews from the Food and Drug Administration (FDA) for studies of 12 antidepressant agents involving 12,564 patients. . . . Among 74 FDA-registered studies, 31%, accounting for 3449 study participants, were not published. Whether and how the studies were published were associated with the study outcome. A total of 37 studies viewed by the FDA as having positive results were published; 1 study viewed as positive was not published. Studies viewed by the FDA as having negative or questionable results were, with 3 exceptions, either not published (22 studies) or published in a way that, in our opinion, conveyed a positive outcome (11 studies). According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive. Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size ranged from 11 to 69% for individual drugs and was 32% overall. (emphasis added).
Melander, et al. also worried (in 2003) that, since “The degree of multiple publication, selective publication, and selective reporting differed between products,” “any attempt to recommend a specific selective serotonin reuptake inhibitor from the publicly available data only is likely to be based on biased evidence.” Without clearer “best practices” for data publication, clinical judgment may be impaired.
Full disclosure of study funding should also be mandatory and conspicuous, wherever results are published. Ernest R. House has reported that, “In a study of 370 ‘randomized’ drug trials, studies recommended the experimental drug as the ‘treatment of choice’ in 51% of trials sponsored by for-profit organizations compared to 16% sponsored by nonprofits.” The commodification of research has made it too easy to manipulate results, as Bartlett & Steele have argued:
One big factor in the shift of clinical trials to foreign countries is a loophole in F.D.A. regulations: if studies in the United States suggest that a drug has no benefit, trials from abroad can often be used in their stead to secure F.D.A. approval. There’s even a term for countries that have shown themselves to be especially amenable when drug companies need positive data fast: they’re called “rescue countries.” Rescue countries came to the aid of Ketek, the first of a new generation of widely heralded antibiotics to treat respiratory-tract infections. Ketek was developed in the 1990s by Aventis Pharmaceuticals, now Sanofi-Aventis. In 2004 . . . the F.D.A. certified Ketek as safe and effective. The F.D.A.’s decision was based heavily on the results of studies in Hungary, Morocco, Tunisia, and Turkey.
The approval came less than one month after a researcher in the United States was sentenced to 57 months in prison for falsifying her own Ketek data. . . . As the months ticked by, and the number of people taking the drug climbed steadily, the F.D.A. began to get reports of adverse reactions, including serious liver damage that sometimes led to death. . . . [C]ritics were especially concerned about an ongoing trial in which 4,000 infants and children, some as young as six months, were recruited in more than a dozen countries for an experiment to assess Ketek’s effectiveness in treating ear infections and tonsillitis. The trial had been sanctioned over the objections of the F.D.A.’s own reviewers. . . . In 2006, after inquiries from Congress, the F.D.A. asked Sanofi-Aventis to halt the trial. Less than a year later, one day before the start of a congressional hearing on the F.D.A.’s approval of the drug, the agency suddenly slapped a so-called black-box warning on the label of Ketek, restricting its use. (A black-box warning is the most serious step the F.D.A. can take short of removing a drug from the market.) By then the F.D.A. had received 93 reports of severe adverse reactions to Ketek, resulting in 12 deaths.
The great anti-depressant debate is part of a much larger “re-think” of the validity of data. Medical claims can spread virally without much evidence. According to a notable meta-researcher, “much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong.” The “decline effect” dogs science generally. Statisticians are also debunking ballyhooed efforts to target cancer treatments.
Max Weber once said that “radical doubt is the father of knowledge.” Perhaps DSM-VI will include a diagnosis for such debilitating skepticism. But I think there’s much to be learned from an insistence that true science is open, inspectable, and replicable. Harvard’s program on “Digital Scholarship” and the Yale Roundtable on Data and Code Sharing* have taken up this cause, as has the work of Victoria Stodden.
We often hear that the academic sector has to become more “corporate” if it is to survive and thrive. At least when it comes to health data, the reverse is true: corporations must become much more open about the sources and limits of the studies they conduct. We can’t resolve the “great anti-depressant debate,” or prevent future questioning of pharma’s bona fides, without such commitments.
*In the spirit of full disclosure: I did participate in this roundtable.
X-Posted: Health Law Profs Blog.
July 14, 2011 at 3:56 pm
Posted in: Bioethics, Culture, Health Law, Law and Psychology, Philosophy of Social Science, Technology
Print This Post
Responses (3)
Daniel S. Goldberg - July 15, 2011 at 9:31 am
Hi Frank,
A typically excellent post. A couple of thoughts leap to mind:
(1) Although it is making news since the publication of Kirsch and Whitaker’s new books, and Angell’s review of them, my strong sense is that a lot of this is relatively old news. We’ve known for quite some time that the only population for whom SSRIs seem to have efficacy above placebo are severely — SEVERELY — depressed persons.
This upsets a lot of people, and rightly so, but I do think it is important not to denigrate the power of placebo pathways, which I hear a lot (“oh that’s just placebo”). As someone who has studied pain for many years, I believe placebos are fabulous things! (The issues as you correctly point out have a lot more to do with the not-insignificant side effect profile and the pernicious and rank influence of COIs).
(2) What’s more interesting to me is that if health care providers have generally been charged with knowledge of these efficacy limitations for some time now, why is the use of such drugs — not to mention antipsychotics — so profligate? Money is one obvious reason, and it is correct, IMO, but it is not the only reason. This is something I am working on, and I actually think it is a lot to do with biological and neurological reductionism, the idea that mental illnesses can be reduced to neurochemistry. This is a category error, as I and many others have written, but what’s also interesting is to trace the influence of historical ideas and conditions on currently dominant narratives of such reductionism. For example, why is the chemical imbalance theory of depression, which is almost unquestionably nonsense on a par with phlogiston, so popular? Why does it show up in so much pharma marketing materials? Why do doctors discuss it so much with their patients (they do)?
I have some theories on this!
(3) And you knew this was coming, didn’t you . . . we don’t want to reduce the complex social problem of mental health and illness to a paradigm of health care. We have very good evidence that depression tracks social gradients, and given the allostatic load hypothesis, it is not difficult to imagine how those marginalized persons experiencing deleterious social and economic conditions and the accumulation of disadvantage would be more likely to experience mental health problems.
So as important as the debate over SSRIs are, and it is important, I think it is critical to steer clear of the medicalization and pharmaceuticalization of society, in which the cause and answer to almost any complex health problem is the absence/administration of medical care. The epidemiologic evidence strongly suggests this is simply not true. I think if we really want to reduce the inequitable burden of mental illness in this society, we need to begin by thinking about social policies that have very little to do with the health care industry, including and especially drugs.
/end rant
(x-posted to Health Law Profs Blog)
Jacqueline Fox - July 16, 2011 at 5:33 pm
Frank, I have a few thoughts on this. Open access to data is important, and it is well known that keeping it secret slows down the development of science, cures, health care advances, etc. I think suppression and manipulation of data also leads to a failure of public trust in all sorts of important endeavors, including evidence-based medicine. But this data is worth a fortune, and it is not hard to see why corporations would behave this way. It has been estimated that the trove of protected scientific data held by the FDA is the single most valuable piece of property that the federal government has. Historically, the FDA has fought hard against any FOIA disclosure of this information, knowing how valuable it is to the pharmaceutical industry. I also think the Bayh-Dole Act bears some blame for this secreting of information, because, prior to that law, data derived from government funded research was made public as a matter of course.
I think Daniel is right that this information about both SSRI effectiveness and data suppression has been around for quite some time. I wrote an article (published in 2007) about drug company behavior and SSRIs, and these questions went back years before that.
I keep looking for some approach to reform that will move scientific knowledge forward, and make what we purport to know more reliable.
Frank Pasquale - July 17, 2011 at 2:58 pm
Many thanks for both comments.
Jacqueline, that’s a valuable broader perspective on the issue. I really do worry, though, that the fundamental model—of the US gov’t effectively “paying” for research by protecting its secrecy—is broken. I think that in the long run, the government should consider a “risk corridors” approach to this research: guaranteeing companies recoupment of most costs, but extracting as a price for that guarantee a) open data and b) limits on profits. I really feel that the profit motive has to be subordinated more to clinical judgments about what are the most important diseases to cure. (John Kay’s book Obliquity suggests there may not even be a trade-off here; he argues that most companies that are primarily committed to a real mission (like curing illness) end up being more profitable and prosperous than those which singlemindedly focus on “shareholder value.” His main examples are from pharma.) I am afraid our politicians will never come to this realization until some truly devastating “Superbug” appears. And by then, of course, it will be too late.
Daniel, I agree, overmedicalization is the larger issue. I linked to an article of mine that suggests an economically driven pathologization of attitudes and work ethics that may be entirely normal. (For example, depressed persons with 3 days of vacation a year may need time off (and time to exercise each day) far more than they need SSRIs.)
I would like to read your anti-reductionism article. I agree that in our climate of opinion, brain-based claims seem to be at the top of the intellectual food chain, followed by economics. I was recently listening to Adair Turner’s Lionel Robbins lecture, where he used Kahneman & Twersky + Frank Knight’s account of risk/uncertainty to trump efficient markets thinking. But of course anyone who’d read Charles Taylor’s critiques of value-neutrality, individualism in social science, and the project of modeling the human sciences on the natural sciences would have grasped the faults of Greenspan-ism/Chicago L&E long ago. (And they would have been treated to that understanding in a much more humane, broad-minded, and holistic way than neuromania (http://www.oup.com/us/catalog/general/subject/Psychology/?view=usa&ci=9780199591343) or behavioral econ offers.)
Leave a Reply