Home > rant, statistics > How to lie with statistics, Merck style

How to lie with statistics, Merck style

July 12, 2012

In the pharmaceutical industry, where companies are making enormous bets with huge money and people’s lives, it makes sense that there are conflicting interests. The companies, who are in charge of testing their drugs for safety and for successful treatment, tend to want to emphasize the good and ignore the bad.

That’s why they are expected to describe beforehand how they are planning to do the tests. It stands to reason that, if they did a thousand tests and then only reported on the best ones, the public would get a biased view of the safety of their products.

For some reason, though, this standard doesn’t seem to be universally followed, and lying with statistics seems to be okay.

The newest example comes from Merck (see Pharmalot article here), which changed its statistical methods on testing Vioxx for Alzheimer’s patients from an intent-to-treat analysis to an on-treatment analysis even though their stipulated plans were the former. And even though the standard in the industry is the former.

Intent-to-treat means you choose people and stick with them, even if they get off the drug for some reason. And on-treatment only counts people that stay on the drug the whole time.

The difference is one of survivorship bias; there may be a good reason someone gets off the drug, and that may be because they got sick, and maybe they got sick because they were taking the drug.

What’s the difference in this case? From the article:

A subsequent intent-to-treat analysis found that as of April 11, 2002, when the FDA approved Vioxx labeling, there were 17 confirmed cardiovascular deaths on Vioxx compared with five on placebo in the same two trials.

With their on-treatment analysis, though, they didn’t see an elevated risk. So as it turns out the actual heart attacks happened a couple of weeks after people got off the pill.

So what happened there? Why were they allowed to change their stipulated method? Why were they allowed to not report their stipulated, gold-standard method? That’s complete bullshit and it must mean that someone at the FDA is either insanely stupid or very rich. Or both.

I’ve written about this issue before, specifically here. Just let me remind you of how we might assess the damage done by Merck through their statistical shenanigans:

Also on the Congress testimony I mentioned above is Dr. David Graham, who speaks passionately from minute 41:11 to minute 53:37 about Vioxx and how it is a symptom of a broken regulatory system. Please take 10 minutes to listen if you can.

He claims a conservative estimate is that 100,000 people have had heart attacks as a result of using Vioxx, leading to between 30,000 and 40,000 deaths (again conservatively estimated). He points out that this 100,000 is 5% of Iowa, and in terms people may understand better, this is like 4 aircraft falling out of the sky every week for 5 years.

According to this blog, the noticeable downwards blip in overall death count nationwide in 2004 is probably due to the fact that Vioxx was taken off the market that year.

Finally, I’d like to reiterate my question, why are pharmaceutical companies allowed to do their own trials?

Categories: rant, statistics
  1. July 12, 2012 at 12:46 pm

    Reblogged this on Guzman's Mathematics Weblog and commented:
    Another rant by one of my favorite bloggers, mathematician Cathy O’Neil (mathbabe). This is really a must read IMO and it illustrates why it is important to understand when statistics are being misused.

    Another good article on the misuse of statistics is “If correlation doesn’t imply causation, then what does?”
    by Michael Nielsen at http://www.michaelnielsen.org/ddi/if-correlation-doesnt-imply-causation-then-what-does/

    Check them both out!


  2. Lior Silberman
    July 12, 2012 at 7:05 pm

    Why would anyone who is not the drug company do a trial that only the drug company stands to profit from?


    • July 13, 2012 at 12:22 pm

      Dear Lior — One suggestion is to have an independent organization that arranges and reports on drug trials, and that is funded by the pharmaceutical industry, but that is not directly commisioned by individual drug companies for individual drug trials.


      • Lior Silberman
        July 13, 2012 at 8:57 pm

        That has occurred to me, of course, but since the studies would have to be funded by the drug companies the organization would be captured by them (even government regulatory agencies tend to be captured by the industries they regulate). The problem is that the drug companies, and everyone they fund, don’t have the incentive to do this right. Consider what happened between the bank regulators and the banks.

        I’d require the drug companies to publish full details of every trial they conduct (regardless of the results, and whether the treatment was later submitted to the FDA for approval or marketed to the public). This way people who have incentive to check the studies could do so.


  3. ael
    July 15, 2012 at 12:53 pm

    If you draw attention to this kind of thing, your email may be monitored: http://www.metafilter.com/117908/In-the-Public-Interest


  1. July 12, 2012 at 5:55 pm
Comments are closed.
%d bloggers like this: