7 JUNE 2007
If you take Avandia for diabetes, or any medicine approved for anything, you perhaps have been alarmed when the media reports some previously undetected danger that might turn a life-saving drug into a killer.
In
Avandia's case, such a report suggested that the drug raises the risk
of heart attacks, apparently bad news for the million Americans who take
it.
Or maybe not -- the evidence is pretty flimsy. The real bad news is that nobody really knows if Avandia is dangerous. For one thing, the government's drug testing and approval process seems incapable of providing such information. For another, the scientific methodology addressing such questions is often foolish and flawed. Not to mention that media reports are often oversimplified, distorted and confused.
Consider the media coverage of the Avandia study reported last month in the New England Journal of Medicine. That analysis showed that Avandia "significantly increased the risk of heart attacks," one newspaper reported. "Researchers also found that the drug boosted the chances of dying of heart disease by 64%," wrote another.
Those writers did not, however, appear to have much of a clue about
the evidence underlying their statements. Their words conflated "significance"
with "statistical significance," a measure not of the importance of the
effect, but an indication of whether an apparent effect is actually due
to chance. 
In fact, the reported increase in deaths was NOT statistically significant (based on the common convention assigning significance to a result with as less than a 5 percent likelihood of being due to chance). The increased heart attack risk was statistically significant, but just barely.
One newspaper report provided raw data, noting that 86 heart attacks occurred among 15,560 patients who took Avandia, while only 72 heart attacks were recorded among the 12,283 given a placebo or some other drug. That makes it sound like many more heart attacks occurred with Avandia. But note that Avandia was taken by more people. A proper comparison would point out that 59 people per 10,000 had heart attacks without Avandia, and only 55 people in 10,000 had heart attacks when using Avandia.
So why all the fuss? It looks like Avandia is a good thing. But that was before the statistical analysis began. These numbers did not come from one study of Avandia (technically, rosiglitazone), but from a combination of dozens of clinical trials, most of them small, with only a few hundred patients.
Small studies are not good ways to estimate risk, because the statistical errors can be large, and a small increase in risk, even if real, might not appear to be statistically significant. So sometimes researchers add a bunch of studies together, performing what is called a "meta-analysis." It's a technical process, involving several mathematical manipulations for combining the studies and deciding how much weight to give to each of them. At the end of that process, the authors of this meta-analysis concluded that rosiglitazone actually increased the risk.
In
principle, meta-analysis is a good idea, because larger numbers of subjects
make the statistical errors smaller. But the statistical tricks are legitimate
only if several criteria are satisfied, and in real life that is almost
impossible.
For one thing, all the studies ever done must be included, unpublished as well as published. And all studies must use the same methods, the same definitions and procedures, to justify lumping them together for statistical tests.
Many unpublished studies were included in the Avandia meta-analysis. But they were not all conducted according to the same rules. In some cases, Avandia was given along with other drugs. Sometimes the non-Avandia group got placebo pills, while in other trials that group received another drug. And there were no common definitions.
"Across the trials, there was no standard method for identifying or validating outcomes; events . . . may have been missed or misclassified," Drs. Bruce Psaty and Curt Furberg wrote in an editorial accompanying the New England Journal report. "A few events either way might have changed the findings."
The study had its strengths, Psaty and Furberg noted. But its weaknesses were substantial, with no data on the relationship between the level of dose and the danger, for instance.
"In this setting, the possibility that the findings were due to chance cannot be excluded," Psaty and Furberg wrote.
Unfortunately, you can't conclude that chance IS to blame, either. It may be that Avandia really does raise the risk of heart attack, and even if the risk is small -- a few extra attacks per thousand people or so -- the consequences are serious when a million people are taking the drug. But the evidence available so far does not establish the risk very convincingly.
The way to tell is not through meta-analysis, but with a well-designed, large trial specifically testing the heart risk. One large trial is under way, but it has problems, too, such as the patients and doctors knowing who is getting Avandia. (An interim analysis of that trial, just published, showed no statistically significant increase in heart attacks or deaths among patients on Avandia.)
Even in large trials, though, the usual statistical methods are suspect, frequently misapplied and misinterpreted. And the Food and Drug Administration's system for approving drugs, and then watching for dangers that appear after approval, does not work very well. Add in the effects of misleading media reports, and the quality of medical information these days seems about as good as the Detroit Pistons' defense against LeBron James.
Hmmm. Maybe if the media devoted as much resources to science and medicine as to sports, the dangers of drugs would be analyzed more reliably.
E-mail: tsiegfried@nasw.org
