What the November, 2009, Breast Cancer Screening Argument Obscured

In November, 2009, a rancorous argument about screening mammography for women aged 40-49 was touched off by the publication of updated guidelines(1), supported by a systematic literature review(2) by the US Preventive Services Task Force (USPSTF).  The guidelines suggested that yearly mammographic screening for women in that age group should not be automatic, but a decision made for individual patients after discussion between the patients and their doctors.  This was based on a critical review of the best available data which suggested that the benefits of screening acrue to only a few patients.  1904 women would have to start screening and continue for multiple rounds to prevent one cancer death over 11-20 years of follow-up.  These benefits had to be balanced against a number of potential risks, including the risks of treatment of cancers that might never behave malignantly, and the at least theoretical risk of generating new cancers through radiation exposure from mammography. 

These seemingly reasonable recommendations generated heated responses.  The debate, to be charitable, seemed to be at its core about how one should weigh benefits and harms in making individual and health policy decisions.  Since different people value different outcomes differently, I was not sure at the time how to make a meaningful contribution to this debate, or whether the debate had to do with the issues we usually discuss on Health Care Renewal.

I should note that the USPSTF guidelines never said "do not screen" women under age 50, or that the government should not pay for such screening.  They did say "the decision to start regular, biennial screening mammography before the age of 50 years should be an individual one and take pateint context into account, including the patient's values regarding specific benefits and harms."  It is hard to see how anyone could argue with that as an expression empowering patients' choices and values.  (For further discussion about how the recommendations were actually modest and reasonable, see Partridge and Winer.)(3)

Now that the dust has settled, it may be useful to reflect further on this.  Doing so suggests that the rancorous US debate mainly obscured rather than illuminated the major issues regarding mammography screening, particularly about our lack of clear evidence from clinical research needed to make the best individual and  policy decisions about mammography. 

It seems to me that the main questions one must answer to make an individual or policy decision about screening mammography are:

Does it improve longevity?

This is not the same as asking whether mammography reduces mortality from breast cancer.  It is theoretically possible that while decreasing mortality due to breast cancer, screening and its downstream consequences increases mortality from other causes.  At least in theory, screening may detect small tumors that would never grow or metastasize.  Treating such tumors could sometimes lead to premature death due to complications of surgery, radiation, or chemotherapy.  Furthermore, screening also involves periodically exposing large numbers of women to radiation, which may sometimes cause new tumors.  So reducing breast cancer mortality does not automatically mean that overall longevity would be improved.

There have been eight major trials of breast cancer screening which included women younger than 50. (See reference 2.)  None demonstrated a statistically significant increase in overall survival (that is, an increase unlikely to have been due to chance alone) due to screening.

Does it reduce suffering, improve functioning or generally improve quality of life?

To my knowledge, no major trial attempted to answer this question.  No such data is mentioned in the USPSTF systematic review.

Do the above benefits outweigh all its potential harms and risks?

So we cannot answer this question, because the benefits that might be most meaningful to patients (overall survival, symptom reduction, functional improvement, overall quality of life improvement) have not been clearly measured.

A Lack of Relevant Evidence

So the USPSTF guidelines, like other relevant guidelines, were based on the evidence that is available.  Since the evidence did not directly answer the most important questions, the guideline writers were left doing the best they could with evidence that only indirectly addressed the main issues.  No wonder they ended up unable to make a clear recommendation, and leaving the decisions to individual discussions, and individual discussions that would necessarily hinge on guesses about the unknown. 

One would think that a big point of discussion about breast cancer screening would be why after eight trials enrolling a total of about 350,000 patients reported over 20 years we still cannot answer the big clinical questions.  A related point for discussion in the US is why only one, and the earliest trial was conducted here.  If we here in the US think breast cancer screening is such a major concern (and we should think so), why have we been unable to mount a single important trial of it since the HIP trial conducted more than 30 years ago?

Instead, the rancorous debate in the US included...

Anecdotes, Some Irrelevant

The press found a number of women who said they would not be alive were it not for screening mammography before age 50.  With all due respect, one cannot tell whether an individual whose tumor was found on screening mammography would still have been diagnosed early enough for succesful treatment in the absence of screening mammography.  (And also with all due respect, we have no idea whether there also are cases of women who died as a result of treatments of tumors that never would have progressed, or cases of women who died of tumors caused by radiation from multiple mammograms.)  Reasoning from single cases when people, diseases, and treatment results vary so much is likely to mislead. 

It is somewhat ironic that some of the cases cited were of women who had breast cancer diagnosed before age 40, even though the debate was supposedly about screening from ages 40-49.  For example, in an inflammatory article that suggested that some "oncologists" might want the USPSTF sent to the prison at Guantanamo Bay, Washington Post editorialist Dana Milbank cited cancer activist Nancy Brinker, who mentioned her sister "whose breast cancer was found with a mammogram at age 37," (and apparently who tragically is no longer alive).(4) 

Going Well Beyond the Evidence

As noted above, no trial has shown that screening mammography for women under 50 increases overall longevity.  We all hope it does, but so far, there is no clear evidence that it does. 

Yet multiple media reports included assertions that screening mammography saves lives.   For example, the breast cancer activist mentioned above said, "mammography saves lives," apparently including mammography under age 50.(4)  An op-ed column by Dr Alan Kaye, chairman of radiology at Bridgeport Hospital, asserted "large, multinational research studies have shown that mammography saves lives in all age groups covered by the current guidelines."(5)  I would challenge him to show me a single such study that found a statistically significant increase in overall survival for patients under 50.  A Texas radiologist stated, "I diagnosed a 40-year old woman with breast cancer last week.  If she had waited 10 years, with pre-menopause breast cancer she would have been dead."(6)  Unfortunately, since she was just diagnosed, how can he be certain that she will survive any given amount of time?  How could he know that the cancer might not have become manifest, absent that single mammogram, later while still treatable? 

I do not want to be too hard on patients who do not appreciate that the outcomes of testing and treatment for breast cancer are not certain.  However, one would hope that physicians would be able to deal with this uncertainty.

Conflicted Opinions

Some of the more strident discourse came from those who may have had financial vested interests in promoting screening mammography.  Fugh-Berman and Bell pointed out numerous "fact-free emotionally charged statements" made by people who appeared to "reading from the same script-book."(7)  They identified that many of the loudest critics of the USPSTF guidelines were affiliated with not-for-profit organizations with impressive names, but also with substantial financial support from corporations that make products used in mammography.  Also, some had personal financial relationships with such corporations. 

An op-ed article by former US Food and Drug Agency (FDA) commissioner Dr Andrew von Eschenbach and Ms Nancy Desmond distorted the USPSTF guidelines to mean "most women should delay screening until they are 50," and claimed that was based on cost concerns, not clinical evidence.(8).  Desmond is the CEO of and von Eschenbach is now a senior advisor to the Center for Health Transformation.  The Center's members include numerous pharmaceutical and device manufacturing corporations, including several that make mammography equipment (e.g., GE Healthcare and Siemens).

Summary

Cancer, especially breast cancer, has major emotional connotations, and can be a difficult issue to deal with from many people.  The conflicting emotions cancer brings out in many patients may understandably affect their physicians, as well as friends and family.  Nonetheless, physicians, other health policy professionals, and health policy experts can serve patients better if they do not allow the patients' understandable affective responses cloud their understanding of the clinical and scientific issues. 

Yet the late 2009 debate in the US about screening mammography included many responses in which emotion seemed to overwhelm reason.  It may also be that some such responses came from people who had vested interests, or whose employers had vested interests that supported the emotional, rather than the reasoned approach.  Meanwhile, no one seemed to acknowledge that a big reason we are still debating this topic is that we have not made the effort or expended the resources to do good trials of sufficient size to answer the questions that need answering.  Of course, such trials might provide answers that would upset some people, or threaten others' incomes.  (As one news article pointed out, mammography is now a $5 billion a year industry in the US.)(6)

So my end of annus horribilis 2009 message on Health Care Renewal is to better serve our patients, from 2010 onward we health care professionals need  to try harder to put evidence and logic ahead of our own emotions, and certainly ahead of our financial self-interest.  

Note that numerous bloggers have taken on this topic, so see posts on Respectful Insolence, GoozNews,  Health Care Organizational Ethics, and the Evidence in Medicine blog.
References

1. US Preventive Services Task Force. Screening for breast cancer: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med 2009; 151: 716-726. [link here]
2. Nelson HD, Tyne K, Naik A et al. Screening for breast cancer: an update for the U.S. Preventive Services Task Force.   Ann Intern Med 2009; 151: 727-737.  [link here]
3. Partridge AH, Winer EP. On mammography - more agreement than disagreement.  N Engl J Med 2009; 361: 2499-2501. [link here]
4. Milbank D. Feeling farther from the finish.  Washington Post, Nov 24, 2009.  [link here] 
5. Kaye A. An alarming retreat on early detection.  Hartford Courant, Nov 25, 2009 [link here]
6.  Jacobson SJ. Dallas-area clinics ignore proposed rules, still push for mammograms. Dallas News, Nov 27, 2009.  [link here]
7. Fugh-Berman A, Bell A. Mammography and the corporate breast.  Bioethics Forum, Nov 24, 2009.  [link here]
8. von Eschenbach A, Desmond N. Government panels can't put price on human life. Associated Press, Nov 24, 2009.  [link here]

The Editorial that Wasn't: Evidence for Systematic Research Manipulation Undetectable by Critical Review

Woe to those of us who have been advocates for evidence-based medicine.  A short description of the evidence-based medicine is medicine whose practice is informed by critical, rigorous review of the best available evidence from clinical research as revealed by systematic search of the published research literature, as well as by the clinician's understanding of biology and the medical and biopsychosocial context, and by the patient's own values.

Evidence-based medicine is based on some key assumptions.  One is that a systematic review will reveal all the results of research studies that are relevant to the issue at hand.  A second is that while the research studies may be flawed and imperfect, they are reported honestly.

Unfortunately, as we have repeatedly discussed, there is more and more evidence that a systematic review will not reveal all relevant results, because research studies may be suppressed, perhaps often, when their results are unfavorable to vested interests.  (Look here for further discussion.)  There is also considerable anecdotal evidence that the design, implementation, and analysis of research studies may be manipulated to make the results more likely to favor vested interests.  (Look here for further discussion.)

An article by Vedula et al just published in the renowned New England Journal of Medicine suggests that such manipulation might be systemic, and that the reporting of manipulated studies may not clearly show what manipulation was done. (1)

To summarize, the authors got access through legal proceedings to internal study protocols and research reports from clinical trials sponsored by Parke-Davis (later merged into Pfizer Inc) of gabapentin (Neurontin) for a variety of clinical problems other than seizures.  Gabapentin was originally marketed as an anti-seizure drug, but Pfizer later "admitted guilt for off-label marketing."  (We discussed the stealth marketing campaign for Neurontin here.)  Vedula et al compared the primary outcome variables specified in the original research protocols, internal research reports, and any publications of the trials' results.  They identified 21 trials, 13 of which were published.  For 12 of the 13, the authors had access to the internal protocol, report, or both.  The main results were:

For 8 of the 12 published trials, there was a disagreement between the definition of the primary outcome in the protocol and that in the published report.... Sources of disagreement included the introduction of an entirely new primary outcome in the published report (in the case of 6 trials); failure to distinguish between primary and secondary outcomes in the published report, even though the protocol did distinguish between them (2 trials); relegation of a primary outcome in the protocol to a secondary outcome in the published report (2 trials); and failure to include in the published report one or more primary outcomes specified in the protocol (5 trials).

Furthermore, it appeared that failure of published articles to clearly and fully report results in terms of the original, pre-specified primary outcome variables occurred when these comparisons were not favorable to gabapentin. As the authors summarized:

Thus, trials with findings that were not statistically significant (P≥0.05) for the protocol-defined primary outcome, according to the internal documents, either were not published in full or were published with a changed primary outcome.

As shown in Figure 3, all the changes that took place between what was specified in the protocol, what was known before publication (as presented in the internal company research reports), and what was reported to the public led to a more favorable presentation in the medical literature of gabapentin's efficacy for unapproved indications.

They concluded:

We are concerned that the reporting practices observed in our analysis do not meet the ethical standards for clinical research or maintain the integrity of scientific knowledge. Fair and honest treatment of patients enrolled in clinical trials of any kind requires full, open, and unbiased reporting. Journal publication, a formalized platform for scientific discourse and dissemination of knowledge, should not be used as a marketing tool for off-label drug use.

Reporting biases such as those we describe here increase the likelihood that interventions will appear to be effective when they are not. Such biases can lead to the omission of negative findings in systematic reviews of intervention effectiveness and in evidence-based guidelines. For example, the 2005 Cochrane systematic review regarding the effectiveness of gabapentin for acute and chronic pain concluded that it is effective on the basis of published findings and should now be updated with the inclusion of unpublished information made available through litigation.

I believe that the article by Vedula et al is particularly important because it shows what appears to be systematic manipulation of the analysis and reporting of multiple clinical trials of the same drug (but for different indications) that had the effect of making the drug appear efficacious when it likely was not. Furthermore, the manipulation was concealed. The published research articles did not completely describe what the intended primary efficacy outcome variables were, did not provide results in terms of these variables, and instead provided results only in terms of variables that were chosen post-hoc as new primary outcome variables.

To address that latter point, I independently reviewed three of the research publications cited by Vedula et al.(2,3,4) All three were noted to have reported significant results favoring gabapentin in terms of a primary efficacy outcome variable that was not identified as such in the corresponding studies' original research protocols or reports (see the supplementary data provided with the New England Journal article.) Per my review, none of the three published articles offered any hint that what they reported as primary outcome variables were not the variables originally chosen in that capacity, nor did they identify what those original primary outcome variables were, or how comparisons made using them turned out.

The implications of the article by Vedula et al are very important. Hence, I was surprised that the article appeared without an accompanying editorial to discuss these implications, and that its publication did not generate much media interest.

So that gives me an opportunity to comment further.

One could start with the implications for evidence-based medicine. As noted above, a short description of the evidence-based medicine is medicine whose practice is informed by critical, rigorous review of the best available evidence from clinical research as revealed by systematic search of the published research literature, as well as by the clinician's understanding of biology and the medical and biopsychosocial context, and by the patient's own values. Evidence-based medicine depends on critical, rigorous review, but the review process is generally done under the assumptions that research publications honestly describe what was done and what its results were. The review process was never designed to detect dishonest reporting or find information that was deliberately concealed. Manipulation of research (design, analysis, and implementation), concealment of that manipulation, and outright suppression of research threaten the foundations of evidence-based medicine. Yet the article by Vedula et al is part of a growing body of evidence that such manipulation, concealment and suppression are widespread, and done to serve vested interests, often commercial.

That is a huge problem for proponents of evidence-based medicine, but also for physicians who want their practices to be based on science, for patients who want their care to be based on science, and for all those in society who see the advancement of medical science as a way to improve peoples' lives.

As senior author Professor Kay Dickersin noted in an interview with Bloomberg "The trouble is, as a scientist, the publication has always been held up to me as the truth. It's the scientific record. What this study indicated is we can't believe that record."

Furthermore, to be a bit more concrete, most physicians, patients, and policy makers depend on what appears to be honest clinical research to make decisions about individual care and health policy. Deliberate and deceitful manipulation of clinical research to favor sponsors' products has likely lead to excessive use of and payments for drugs and devices that are less effective than advertised, if not useless or dangerous. Thus, it is likely that such manipulation is partially responsible for ever increasing health costs and poor health outcomes.

Finally, we need to start thinking about how we can detect and compensate for manipulation of clinical research in the past, and deter such manipulation in the future. One possible deterrent would be, as was noted by Vedula et al, detailed clinical trials registries that contain complete information about trial protocols.  For this to be effective, there need to be mechanisms to assure compliance, and penalties for non-compliance.  Moreover, since clinical research is now global, the registries must have global scope, and enforced assurance of compliance must also be global.

Registries might decrease future manipulation and suppression of research.   No one has suggested, as far as I know, a systematic way to detect and correct for previous manipulation. It would require a major, global investigative effort to uncover manipulation, and it would be a major scientific and policy endeavor to reveal most suppressed research and correct most manipulations.

However, before anything is done, patients, physicians and policy makers must acknowledge and understand the problem.  Yet it seems that even discussing these may be topics that are very uncomfortable for some of us. The longer we shrink from addressing them, however, the worse will be the results for patients, physicians, science and society.

References


1. Vedula SS, Bero L, Scherer RW, Dickersin K. Outcome reporting in industry-sponsored tirals of gabapentin for off-label use. N Engl J Med 2009; 361: 1963-1971. Link here.


2. Mathew NT, Rapoport A, Saper J et al. Efficacy of gabapentin in migraine prophylaxis. Headache 2001; 41: 119-128. Link here.


3. Vieta E, Goikolea JM, Martinea-Aran A et al. A double-blind, randomized, placebo-controlled, prophylaxis study of adjunctive gabapentin for bipolar disorder. J Clin Psychiatr 2006; 67: 473-477.  Link here.


4. Caraceni A, Zecca E, Bonezzi C et al. Gabapentin for neuropathic cancer pain: a randomized controlled trial from the gabapentin cancer pain study group. J Clin Oncol 2004; 22: 2909-2917. Link here.

Failing to Report Adverse Effects of Treatments

We have frequently advocated the evidence-based medicine (EBM) approach to improve the care of individual patients, and to improve health care quality at a reasonable cost for populations. Evidence-based medicine is not just medicine based on some sort of evidence. As Dr David Sackett, and colleagues wrote [Sackett DL, Rosenberg WM, Muir Gray JA, Haynes RB, Richardson WS. Evidence-based medicine; what it is and what it isn't. BMJ 1996; 312: 71-72. Link here. ]

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.

One can find other definitions of EBM, but nearly all emphasize that the approach is designed to appropriately apply results from the best clinical research, critically reviewed, to the individual patient, taking into account that patient's clinical characteristics and personal values.

When making decisions about treatments for individual patients, the EBM approach suggests using the best available evidence about possible benefits and harms of treatment, so that the treatment chosen is most likely to maximize benefits and minimize harms for the individual patient. The better the evidence about specific benefits and harms applicable to a particular patient, the greater will be the likelihood that a particular decision based on this evidence will result in the best possible outcomes for the patient.

A new study in the Archives of Internal Medicine focused on how articles report adverse effects found by clinical trials. [Pitrou I, Boutron I, Ahmad N et al. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med 2009; 169: 1756-1761. Link here.] The results were not encouraging.

The investigators assessed 133 articles reporting the results of randomized controlled trials published in 2006 in six English language journals with high impact factors, that is, the most prestigious journals, including the New England Journal of Medicine, Lancet, JAMA, British Medical Journal, and Annals of Internal Medicine. They excluded trials with less common designs, such as randomized cross-over trials. The majority of trials (54.9%) had private, or private mixed with public funding.

The major results were:
15/133 (11.3%) did not report anything about adverse events
36/133 (27.1%) did not report information about the severity of adverse events
63/133 (47.4%) did not report how many patients had to withdraw from the trial due to adverse events
43/133 (32.3%) had major limitations in how they reported adverse events, e.g., reporting only the most common events (even though most trials do not enroll enough patients to detect important but uncommon events).

The authors concluded, "the reporting of harm remains inadequate."

An accompanying editorial [Ioannidis JP. Adverse events in randomized controlled trials: neglected, distorted, and silenced. Arch Intern Med 2009; 169: 1737-1739. Link here] raised concerns about why the reporting of adverse events is so shoddy:

Perhaps conflicts of interest and marketing rather than science have shaped even the often accepted standard that randomized trials study primarily effectiveness, whereas information on harms from medical interventions can wait for case reports and nonrandomized studies. Nonrandomized data are very helpful, but they have limitations, and many harms will remain long undetected if we just wait for spontaneous reporting and other nonrandomized research to reveal them. In an environment where effectiveness benefits are small and shrinking, the randomized trials agenda may need to reprogram its whole mission, including its reporting, toward better understanding of harms.

Pitrou and colleagues have added to our knowledge about the drawbacks of the evidence about treatments that is publicly available to physicians and patients when making decisions about treatment. Even reports of studies with the best designs (randomized controlled trials) in the best journals seem to omit important information about the harms of the treatments they test.

It appears that the majority of the reports that Pitrou and colleagues studied received "private" funding, presumably meaning most were funded by drug, biotechnology, or device companies and were likely meant to evaluate the sponsoring companies' products. However, note that this article did not analyze the relationship of funding source to the completeness of information about adverse effects.

Nonetheless, on Health Care Renewal we have discussed many cases in which research has been manipulated in favor of the vested interests of research sponsors (funders), or in which research unfavorable to their interests has been suppressed. Therefore, it seems plausible that sponsors' influence over how clinical trials are designed, implemented, analyzed and reported may reduce information about the adverse effects of their products reported in journal articles. Trials may be designed not to gather information about adverse events. Analyses of some adverse events, or some aspects of these events may not be performed, or if performed, not reported. The evidence from clinical research available to make treatment decisions consequently may exaggerate the ratios of certain drugs' and devices' benefits to their harms.

Patients may thus receive treatments which are more likely to hurt than to help them, and populations of patients may be overtreated. Impressions that treatments are safer than they actually are may allow their manufacturers to overprice them, so health care costs may rise.

The article by Pitrou and colleagues adds to concerns that we physicians may too often really be practicing pseudo-evidence based medicine when we think we are practicing evidence-based medicine. We cannot judiciously balance benefits and harms of treatments to make the best decisions for patients when evidence about harms is hidden. Clearly, as Ioannidis wrote, we need to "reprogram." However, what we need to reprogram is our current dependence on drug and device manufacturers to pay for (and hence de facto run) evaluations of their own products. If health care reformers really want to improve quality while controlling costs, this is the sort of reform they need to start considering.

NB - See also the comments by Merrill Goozner in the GoozNews blog.

A New Perspective on Evaluating the Effects of Financial Conflicts of Interest on Research

I just posted about an article from the August issue of the Journal of Epidemiology and Community Health (link here, requires subscription.) Health Care Renewal readers may want to peruse this issue, which has some very interesting articles on conflicts of interest and related issues in research.

In particular, an article by Prof Sander Greenland offers a fresh discussion based on perspectives from epidemiology, statistics, and cognitive psychology of the effects of financial conflicts of interest (COI) on (clinical, epidemiological, and health services) research (Greenland S. Accounting for uncertainty about investigator bias: disclosure is informative. J Epidemiol Community Health 2009; 63: 593-598. Link here, requires subscription.)

Since only subscribers can easily get this article, and because of its importance, let me summarize its main points and provide appropriate quotations.

Financial COI as Causing One Type of Investigator Bias

Greenland views financial conflicts of interest as a source of one kind of investigator bias, defined as "the biasing of study results towards results expected or desired a priori by the investigators." He believes that this is a particularly important kind of bias, "the pool of reported study results may be affected seriously by prejudices and vested interests of investigators and sponsors. Such incidents suggest that investigator bias can be larger than any other bias, and may often encourage adoption of dubious or even deadly treatments."

Focus on Effects of the Bias, Rather than Intent

Greenland posited that the issue here is not intent,

Investigator bias may be intentional or subconscious, may be of venal or noble origin, and may manifest in outright fraud or in subtle fallacies held true by the investigator. It may reflect no more than the influence of sincere wishes or beliefs of the investigator or their research community on the way findings are interpreted, or it may represent premeditated attempts to mislead.

The danger is that raising the issue of intent may lead to emotional arguments rather than sober thinking about what effect conflicts of interest should have on interpreting study results,

here we encounter an emotional minefield of indignation driven by fears of prejudice in and abuse of such priors [that is, beliefs about the effects of investigator bias due to conflicts of interest.] Similar objections meet proposals to down-weight studies that produce results favourable to sponsors or other parties with financial ties to the investigators.

Rather, the issue is that conflicts of interest may indicate a high risk that a study is biased, and obviously, biased in a certain direction, "we expect investigator bias, if present, to fall in the direction favouring their or their sponsor’s financial interest."

An Approach to Evaluating the Effects of Investigator Bias due to Financial COIs

Greenland pointed out that it may be difficult to account for the effects of investigator bias due to COIs by just being careful in the evaluation of the report of a given research study, "incomplete documentation of ad hoc design and analysis decisions makes investigator bias especially difficult to detect and deal with." Furthermore, "one cannot detect unreported biasing judgements without having the data and the time to reanalyse it."

Thus, the challenge is to get accurate information about COIs, and then use it to accurately evaluate a particular study. The realities of life in the often conflicted world of medical academia means that conflicts of interest may generate a large amount of bias. Greenland suggested that this

reflects a harsh reality. In research funded or conducted by parties with a direct financial stake in the results, there may be an implicit threat of funding withdrawal (or worse) if results presented do not favour the sponsor’s interest or conform to their perceived needs for marketing and litigation.

Furthermore, he noted,

Sponsored investigators have been sued by their corporate sponsors in efforts to block publication of undesirable results. Investigators with no corporate ties have faced interference from corporations upset with their findings. Not all investigators choose to fight such pressure, which leaves them the option to either not publish unfavourable results or else not produce such results. Some go so far as to guarantee their sponsor that the results of their proposed study will be helpful.10 Meanwhile, unfavourable results from research conducted within a company may languish as in-house memos that can be obtained only via litigation, and then only if the court orders their release.

However, information provided about COIs may not be complete or accurate.

Once we accept the reality that disclosure data contain relevant statistical information, we can turn to the quality of information ascertainable from available reports.

There are reports of investigators disclosing only a small fraction of their financial support from industry, even when required to give a full account by university and granting-agency rules. There is little reason to expect more thorough disclosure in journal articles. Full disclosure is often evaded via technicalities, such as payment of non-descript consulting fees that are not explicitly earmarked for the article and thus not deemed reportable, even when they may be for time spent on the database from which the article is generated. If reported at all, such funds may be described only by phrasing such as 'author X has served as a consultant (or expert) for company Y', not that the payments for such services were made with the understanding that they would support time spent on the study being reported.

The frequency of under-reporting and deceptive reporting is far from known. Most incidents come to light only when investigation or litigation occurs, and those discoveries are not always reported in public venues. Other devices for undermining disclosure requirements include funds channelled through innocuously titled foundations or institutes set up specifically to conceal the ultimate funding source. Detecting such events and devices may demand investigative effort beyond the resources of almost all readers.

However, while there may be considerable false negative information about COIs, false positive are unlikely, "In contrast, it is hard to imagine that many articles report funding from a financial stakeholder when in fact there was none, so that misreporting may be limited to under-reporting."

Conclusions and Future Directions

Greenland concluded that investigator bias is very important.

Investigator bias has the potential to overwhelm all other biases. It can become the dominant force in contexts (such as expert reports and testimony for litigation) in which the restraints imposed by editors and peer review are absent. Thus, because of its importance, investigator bias should not be dismissed as unapproachable, any more than we should give up research on what seem to be hopelessly mysterious diseases.

He reminded us that not all investigator bias is due to financial COIs, although that should not be used to deny the importance of bias due to financial COIs,

Although I have focused on distortion from financial input, I have no doubt that ideological commitment can be just as distortive. Indeed, dealing with ideological and other psychological sources of bias (such as commitment to earlier methods and conclusions) deserves to be on the research agenda for bias analysis.

He suggested some directions for the future,

In approaching these problems scientifically, I suspect aid will be needed from those branches of psychology that deal with distorted thinking among normal individuals and groups.

Openness of journals to unpopular research and strenuous debate remains the primary line of defence against information abuse and distortion. Such openness is not universal, and thus it seems likely that proposals to deal with investigator bias will raise a hornet’s nest of objections and questions. Perhaps the essential first step (if not major hurdle) is getting used to the idea that there may be good arguments for including the possibility of investigator bias in an uncertainty assessment. The demand for thorough financial disclosure may then seem no different from demanding hard ('objective') details of design and analysis to aid assessments. Like other psychological constructs, ideological biases will be far more difficult to evaluate but may nonetheless have hard correlates that signal their presence if not magnitude, such as attempts to suppress or boycott opposing views.

I highly recommend that those with access to this journal read Prof Greenland's article in full, and see the other related articles in this issue. In particular, Prof Greenland's article, and the article from the German chapter of Transparency International provide some clear thinking as a counter to the illogic that is now appearing as the backlash by the conflicted against the "pharmascolds" intensifies.

Pseudo-Evidence Based Medicine Threatens Health Care Reform Based on "What Works"

As I posted yesterday, the increasingly noisy debate about health care reform in the US has not dealt much with the issues we often discuss on Health Care Renewal. These include problems in how health care organizations are led which threaten physicians' and other health care professionals' core values using tactics including perverse incentives, deception, and intimidation.
Last night, however, President Obama held a news conference mostly devoted to health care issues, in which he stressed the importance of changing not just how health insurance works, but how health care decisions are made. As Newsweek's "The Gaggle" blog reported,

Can I guarantee that there are going to be no changes in the health care delivery system? No. The whole point of this is to try to encourage changes that work for the American people and make them healthier. The government already is making some of these decisions. More importantly, insurance companies right now are making those decisions. And part of what we want to do is to make sure that those decisions are being made by doctors and medical experts based on evidence, based on what works, because that's not how it's working right now.

So what the President seems to be advocating is making health care more evidence-based, perhaps in the formal sense of evidence-based medicine.

As a card-carrying evidence-based medicine advocate, I certainly agree, but let me reiterate that evidence-based medicine is not just medicine based on some sort of evidence. As Dr David Sackett, and colleagues wrote [Sackett DL, Rosenberg WM, Muir Gray JA, Haynes RB, Richardson WS. Evidence-based medicine; what it is and what it isn't. BMJ 1996; 312: 71-72. Link here. ]

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research. By individual clinical expertise we mean the proficiency and judgment that individual clinicians acquire through clinical experience and clinical practice. Increased expertise is reflected in many ways, but especially in more effective and efficient diagnosis and in the more thoughtful identification and compassionate use of individual patients' predicaments, rights, and preferences in making clinical decisions about their care. By best available external clinical evidence we mean clinically relevant research, often from the basic sciences of medicine, but especially from patient centred clinical research into the accuracy and precision of diagnostic tests (including the clinical examination), the power of prognostic markers, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens.

Furthermore,

Evidence based medicine is not 'cookbook' medicine. Because it requires a bottom up approach that integrates the best external evidence with individual clinical expertise and patients' choice, it cannot result in slavish, cookbook approaches to individual patient care. External clinical evidence can inform, but can never replace, individual clinical expertise, and it is this expertise that decides whether the external evidence applies to the individual patient at all and, if so, how it should be integrated into a clinical decision.

One can find other definitions of EBM, but nearly all emphasize that the approach is designed to appropriately apply results from the best clinical research, critically reviewed, to the individual patient, taking into account that patient's clinical characteristics and personal values.

So far, so good. I believe the proper application of "real" (as described above) evidence-based medicine has the potential to improve patient outcomes while moderating health care costs. However, we have pointed out how problems arising from concentration and abuse of power in health care threaten the evidence-based medical ideal.

First, there are major problems with the development of the sort of clinical research evidence required by the EBM process. We have discussed how clinical research is frequently manipulated by those with vested interests in producing results that favor the products or services that they sell. The critical review process inherent in EBM is meant to cope with less than optimally designed and implemented research. However, the process was designed to cope with honest mistakes and inevitable trade-offs, not deliberate manipulation by vested interests.

Worse, we have discussed how vested interests may engineer the suppression of research when manipulation fails to produce the desired results. The EBM process assumes that the research on which decisions should be based is an unbiased sample of research that was done (and done to advance science, not commercial or ideological interests). When research whose results are unwanted by vested interests is suppressed, the resulting distortion of the evidence base may irretrievably bias the EBM process.

Finally, we have posted about how vested interests have distorted the discussion, dissemination, and teaching of the results of clinical research. They may develop systematic stealth marketing campaigns, often employing supposed "key opinion leaders," who are paid on the side by marketers, and using "medical education and communication companies"as marketing fronts whose publication strategies include deceptive tactics such as "ghost-writing."

Thus, a rising tide of "pseudo-evidence based medicine" threatens to overwhelm even the most conscientious physicians trying to practice evidence-based medicine.

So, while I applaud President Obama's advocacy of reforming health care to emphasize what the best evidence suggests really works, I do not think this effort will get far unless we deal with the rising tide of pseudo-evidence based medicine. As a minimum, we need full and detailed disclosure of all the relationships among vested interests and medical research and education, and a much greater role for research and education that is not subsidized by corporations bent on using research and education to market their products and services.

More Documents About the Selling of Seroquel Show How Research Was "Subordinated to Commercial Goals"

We posted earlier this year about how documents produced on discovery and recently unsealed during litigation suggested how AstraZeneca handled clinical research data in the marketing of its atypical anti-psychotic drug Seroquel (quetiapine). A new crop of documents has just been released, providing yet more insights, as reported by the St Petersburg (Florida) Times:

Behind the scenes at the global pharmaceutical company AstraZeneca, the team in charge of the blockbuster antipsychotic Seroquel had one mission: make the multibillion-dollar seller even bigger.

To that end, internal company documents released Wednesday show how the British drugmaker hid unfavorable study results, promoted unapproved uses and even considered pitching the drug as less likely to lead to suicidal thinking than competitors'.

In particular,

Documents suggest Seroquel studies were repeatedly subordinated to commercial goals.

When a study in 2002 failed to show that Seroquel's sustained release formula was any more effective than a placebo in treating schizophrenia, orders from the top were to keep the results 'in strictest confidence.'

When a scientist in England wanted to study the weight gain in rats on Seroquel, AstraZeneca declined to fund the research, saying 'we could wind up with results that are not clearly advantageous.'

In one discussion, it was suggested that authors of potentially helpful research reports who raised too many questions, slowing publication, should be asked to step down.

A researcher who pressed for results of an unfavorable trial was rebuffed for weeks before being given 'three or four sentences describing high-level results.'

AstraZeneca marketers were jealous of what they saw as competitor Lilly's ability to cast questionable study results in a positive light. 'They (Lilly) are able to spin the same data in many different ways through an effective publications team,' according to a 2003 memo. 'Negative data usually remains well hidden.'

As Seroquel's sales soared, documents reflect an ongoing struggle between the safety and marketing teams over the potentially damaging issue of weight gain. In 2000, the company's scientists said data did not support the marketing claim that Seroquel resulted in only 'limited' weight gain. Close to 23 percent of the people who took the drug gained more than 7 percent.

Despite the safety team's objections, the word 'limited' remained on Seroquel's label for two more years.

This is a reminder how beleaguered we advocates of evidence-based health care (EBHC) have become. The idea of EBHC was that health care decisions for individual patients, and policies for groups of patients ought to be guided by critical review of the best available evidence from clinical research, guided by knowledge of biology and the biopsychosocial context of health, and informed by patients' values and preferences. The idea still makes sense to me, but it only works if physicians, patients and policy-makers have access to an unbiased sample of clinical research studies, so that studies with are not selectively suppressed to support vested interests. Although critical review can account for inevitable trade-offs, compromises, and errors in how studies are designed, implemented, and analyzed, the clinical epidemiological methods it uses are really not designed to root out falsehoods and deliberate deception.

However, the ongoing story of Seroquel, and many other cases discussed on Health Care Renewal suggest that when clinical research is sponsored by those who can profit from the product or service it evaluates, that research is prone to suppression and manipulation. Although I believe there are many honest scientists who work for pharmaceutical, biotechnology, device, health information technology and other health care corporations, it seems they often have to answer to marketers whose only goal is to sell more product.

As long as clinical research is sponsored and run by the people who can profit directly from selling the products and services the research is meant to evaluate, the ideal of evidence-based health care becomes less attainable.

As we have said before, suppression and manipulation of research amounts to post-hoc abuse of research subjects who volunteered their participation believing that it would advance science and health care.

Furthermore, suppression and manipulation of research can deceive physicians into prescribing tests and treatments that will fail to help, or even harm patients, and deceive patients into thinking that they are getting the best possible tests and treatments, when, again what they are getting is ineffective or even harmful.

In my humble opinion, there is an increasingly strong argument that clinical research should not be controlled, and probably should not be done at all by organizations with vested interests in the research producing results favorable to their products.

BLOGSCAN - How US Medicare Will Pay for Cancer Drugs

On the GoozNews blog, Merrill Goozner dissects the recent decision by the US Center for Medicare and Medicaid Services (CMS) to make more lenient the criteria it uses to decide which off-label anti-cancer drugs it will pay for. Now any drug rated favorably in at least one of several drug compendia will be approved. Yet it turns out that intricate, but important conflicts of interest may affect how drugs get positive reviews in some of the compendia. It does seem that nearly every piece of our fiendishly complex health care system is affected by conflicts of interest. To put the most positive interpretation on it, most US government agencies that deal with health care seem to have operated up to now in ignorant bliss when it comes to these conflicts.

NB: see also the post by Dr Howard Brody on the Hooked: Ethics, Medicine and Pharma blog, and the links he provides.

The Perils of Contract Research Organizations Out-Sourcing Clinical Trials

The St Petersberg Times last week published an investigative series on the out-sourcing of clinical trials to India, which raised important questions about the quality of data they produce.

Mary K. Pendergast, a former FDA deputy commissioner, said identifying a dangerous product is difficult enough. It's considerably trickier to find fraudulent clinical trial data, which could lead to the approval of dangerous drugs years later.

'It's much more time-consuming and extraordinarily tedious,' said Pendergast, who plowed through such data when she was prosecuting doctors doing drug studies in the United States. 'It's especially hard if the trial is taking place in a different country.'

Particularly when that country has a reputation for cutting corners.

The article quoted a number of people familiar with the clinical trials scene in India who suggested that corners indeed were being cut.

Amar Jasani:

In the burgeoning clinical trial business, says Amar Jesani, a doctor and medical ethicist in Mumbai, every layer of oversight is compromised by cash, and independent monitoring is nonexistent. He has resigned from supposedly independent ethics committees that rubber-stamp drug companies' proposals and overrule any objections.

Dr Arun Bhatt:

Dr. Arun Bhatt is president of Clininvent, a contract research organization in Mumbai that is running 11 trials involving 1,000 patients. He worries that Indian doctors ignore patients' reactions to experimental medicines, missing critical clues about a drug's safety.

'Most sites are not used to recognizing serious adverse events, so they are underreported,' he said with an attitude of resignation. 'Either they don't recognize, don't realize or are afraid to report this information.'

Dr Nandini Kuman:

Dr. Nandini Kumar retired in June as deputy director general of the Indian Council of Medical Research, the equivalent of the U.S. National Institutes of Health. She now works as a consultant for the agency in New Delhi, teaching doctors how to run ethical trials. Kumar is stunned by their ignorance of internationally recognized standards for conducting drug studies.

Asked what aspect of good clinical practice most surprised her students, the gray-haired, sari-clad Kumar snapped, 'Everything.'

'There are efforts to put rules in place, but at the same time there are people who just want to get the extra dollars or perks like trips abroad from doing drug trials....'

Dr S P Kalantri:

Dr. S.P. Kalantri has conducted trials for global pharma at the government hospital in Sevagram, a small town in central India. But he said he has pulled back from doing the studies.

'It's difficult to explain the complexities of trials to study participants,' Kalantri said of the hospital's mostly poor, illiterate patients. 'I think many investigators tend to take their patients for a ride. And there's an abysmal lack of know-how about clinical research among investigators.'

Dr C M Gulhati:

Dr. C.M. Gulhati, editor of an Indian medical journal, Monthly Index of Medical Specialties, said authorities cannot cope with the tsunami of trials.

'India's drugs controller general's office is both understaffed and incompetent,' he said, citing a case where the agency claimed it reviewed an 800-page trial protocol in just five days. 'How is that even possible?'

Gulhati, who fights unbridled drug testing from a dim and cluttered office above a busy Delhi shopping plaza, reeled off a litany of troubled trials:

In 2003 in Hyderabad, an unregistered study of a heart attack drug that resulted in six deaths.

In 2004 in Delhi, a first-in-human trial of a new suturing device on 13 patients without regulators' approval.

Last year, India's decision to become the only nation to allow domestic drug maker Sun Pharmaceutical Industries to market the anticancer drug Letrozole for infertility in women despite the drug's originator, Novartis, warning that it may cause fetal harm and should be used only in postmenopausal women.

And in October in Bangalore, the death of a baby during the testing of a new Wyeth vaccine.

But doing trials in India, and other less-developed countries is far cheaper and faster than doing them in developed countries. Contract research organizations (CROs) have figured this out.

Contract research organizations keep tabs on trials with regular audits. Site management organizations put an extra set of eyes in the doctor's office. Both entities dangle promises of FDA-ready studies in half the time, at 30 to 60 percent the cost in the United States. The secret? Cheap help and fast patient recruiting.

Quintiles, the world's largest contract research organization, boasts of enrolling 50 patients with diabetes in one month in India and 204 infants for a vaccine study in three days, far faster than possible in the States. In a brochure, Quintiles sums up India's allure: 'It's practically a paradise for conducting clinical trials.'

In fact, on Quintiles current web-site is an offer related to Phase II and III trials for

Accelerated Start-up and Recruitment worldwide, through a global network of investigators that includes many in nontraditional regions with large clusters of patients and few competing trials.

We have noted before that the limited oversight of CROs, which are responsible for an increasing proportion of industry-funded trials of drugs and devices, makes possible poor quality work, hence unreliable data (see most recent post here and links backward). The threat to data integrity from too hasty, too profit driven clinical research done by CROs is the third of the three major threats to the integrity of the clinical research data base. The other two, which are discussed frequently on Health Care Renewal, are suppression of clinical research, and manipulation of research design, implementation, and analysis and dissemination.

In combination, they threaten the well-intentioned and idealistic goals of evidence-based medicine, to improve clinical care by basing decisions not only on knowledge of biology and the values of patients, but on critical review of the best available clinical evidence. But given that we have become increasingly skeptical, if not outright distrustful of the integrity of the evidence, the alternative ways of making clinical decisions, basing them on theoretical (but possibly incorrect) knowledge of biology, on authority, or on tradition to me are much worse.

Doctors and patients should be vehement about the need for carefully performed clinical trials free of manipulation meant to benefit vested interests, reported honestly even if their results offend vested interests.

Questions of Benefits vs Risks for the UPMC Liver Transplant Program

Last week, the Wall Street Journal published an investigative report on the liver transplant program at the University of Pittsburgh Medical Center which provided a troubling view of some the issues affecting US health care.

Let me summarize the main points, as I would like to organize them, using quotes from the article, and my parenthetic comments.

Liver Transplantation Strategies at UPMC were Aggressive and Risky

Earlier this decade, UPMC made an aggressive bid to reclaim its leadership by hiring an innovative surgeon named Amadeo Marcos, who promised to double the number of liver transplants the hospital did.

Dr. Marcos delivered on his pledge. In doing so, however, he resorted to practices that some colleagues found questionable.

These practices included:

Lowering Standards for Donor Livers

To overcome a perennial shortage of organs, he used more livers from older donors.

Also,

A shortage of transplantable organs from cadavers is a perennial constraint on the number of liver transplants. Dr. Marcos overcame this in part by using organs from so-called expanded-criteria donors -- deceased people who had been older or sicker than preferred liver donors.

In the 2½ years before Dr. Marcos joined UPMC, the average age of its deceased liver donors was 41, according to UNOS. By 2003, it was 47, or nine years above the national average.

And while in 2000 and 2001, UPMC used an average of only 10 livers a year from patients older than 65, it used 45 in 2003.

Performing Transplants on Less Sick Patients (Who Are Less Likely to Benefit)

He transplanted some of these into relatively healthy patients for whom the risk-reward calculation was less certain.

In particular,

Dr. Marcos put some of these organs into patients who were in the early stages of liver disease, say Dr. Fung and Howard Doyle, who then worked in UPMC's transplant intensive-care unit. These were patients, they say, who sometimes didn't need a transplant.

'For the first time in years, we had people dying on the operating table or in the ICU,' says Dr. Doyle, now director of surgical critical care at Montefiore Medical Center in New York. At times, according to him, patients healthy enough to walk into the hospital before being transplanted died 'because they had a high-risk liver put into them.'

Data from the Scientific Registry of Transplant Recipients show that during Dr. Marcos's time at UPMC, 30 liver recipients died within two days of surgery. That was a death rate of 2.4%, versus a national average of 1.6%.

Also

Liver patients are ranked by how advanced their disease is. Based on a series of blood tests called MELD, scores range from 40 for the sickest to six for the healthiest. Most experts now believe the risks of a transplant generally outweigh the benefits for patients with MELD scores of 14 or lower.

During Dr. Marcos's nearly six years at UPMC, it performed 441 liver transplants on patients with scores of 14 or lower, according to UNOS. That was 35% of the liver transplants performed during his tenure, and compares with fewer than 7% in the 2½ years before he arrived.

Using Live Donors (Who Donate Part of Their Livers, and Also Are At Risk From the Procedure)

He used partial livers from living donors, and then understated complications from the controversial procedure.

In particular,

Dr. Marcos sharply increased the number of transplants from living donors. In these, part of the liver of a healthy person is cut off and grafted into a sick patient. If all goes well, both pieces eventually grow to normal size. The procedure is controversial because it could be risky for the otherwise healthy donor.

UPMC did 150 such surgeries while Dr. Marcos was there, according to UNOS. No donors died. However, in 69% of the cases, the recipient had a MELD score of 14 or lower -- suggesting that UPMC was putting some living donors at risk to do transplants on patients in which the risks of the operation may have outweighed the benefits.

In addition, Dr Thomas Starzl, the liver transplantation pioneer for whom the UPMC center is named,

became suspicious of the low complication rates Dr. Marcos was reporting in adult living-donor liver transplants, say people familiar with the matter. In a textbook Dr. Marcos co-wrote, he said UPMC's rate of serious complications was zero for donors and 34% among a subset of recipients.

Dr. Starzl reviewed the 121 transplants UPMC had done involving removal of the donor's right lobe, a typical procedure in adult-to-adult living-donor liver transplants. Dr. Starzl's finding, according to people with knowledge of it: Though recipients' survival rate was only slightly lower than the national average, 60% of the recipients suffered life-threatening complications, ranging from bile-duct leaks to blood-supply problems -- nearly double the rate Dr. Marcos reported.
Dr. Starzl raised his concerns with UPMC chief Mr. Romoff and other officials, including the head of the department of surgery, Timothy Billiar, say the people familiar with the situation.

A tense six-month standoff ensued. Dr. Starzl, worried that UPMC was covering the matter up, sent his findings to a medical journal, according to people familiar with the events. Dr. Billiar asked it not to publish, on the ground that Dr. Starzl hadn't obtained patient authorization to collect the data. Dr. Billiar says that Dr. Starzl's paper would have jumped the gun on a peer-reviewed internal study he had requested from another surgeon, Wallis Marsh.

UPMC and Dr. Starzl compromised: Dr. Starzl would wait for the internal study, which would be reviewed by Pierre-Alain Clavien, a Zurich surgeon who pioneered a scale to measure complications in living-donor liver transplants. UPMC's final conclusions would be published.

In January, Dr. Marsh and Dr. Clavien confirmed Dr. Starzl's finding of a 60% rate of serious complications among recipients, documents seen by The Wall Street Journal show. The review also concluded that about 10% of the living donors had suffered serious complications, belying Dr. Marcos's claim that this number was zero.

Liver Transplantation Strategies at UPMC were Expensive

Hospitals charge $400,000 to $500,000 for a liver transplant. UPMC's transplant program produced $130 million of revenue in its latest fiscal year.

There Was No Evidence that the Liver Transplantation Strategies at UPMC Provided Benefits Outweighing Their Potential Harms

The WSJ article quoted many UPMC doctors, and gave Mr Paul Wood, the UPMC Vice President for Public Relations, multiple chances to rebut the articles' findings. Mr Wood did declare, "our core mission is nothing less than providing the best and most appropriate care for patients."

However, no one quoted in the article was willing to assert, much less justify, a claim that the aggressive tactics used during Dr Marcos' time at UPMC provided benefits that outweighed their risks, including in some cases their risks to donors as well as patients.

Another news article on the UPMC controversy published by the Pittsburgh Tribune-Review quoted Dr James Trotter, an investigator in the Adult-to-Adult Living Donor Living Transplant Cohort, did state "there's a survival advantage for people who undergo live-donor liver transplant versus waiting on the transplant list."

A quick look at the literature revealed an article from the UPMC program reporting a case-series of patients who received transplants from living donors.(1) The article showed that their survival rate was comparable to that for patients who received cadaver transplants. However, it did not include a control group, and therefore could nor provide even weak evidence that outcomes for patients receiving transplants from living donors are superior to those receiving them from cadavers. In addition, there is evidence that the risks to living donors are clinically significant. For example, a systematic review suggested that donor mortality may be more than 0.2%, and donor morbidity ranged from 0% to 90% in several studies, with a median of 16%.(2)

Thus, no defender of the strategies previously used at UPMC, nor any clinical research that I found provided strong evidence that the benefits of the use of living donors outweighed their risks to the donors, or to the recipients. Furthermore, no one put forward any evidence that the aggressive approach pushed by Dr Marcos generally provided benefits to patients that either exceeded those of a more conservative approach, or outweighed its apparent risks.

Perverse Incentives Favored an Aggressive, Risky Approach

As I noted above, the UPMC liver transplant program brought in a large amount of money, especially considering that the program provided transplants to nver more than 300 patients a year, according to a chart provided in the WSJ article. This enabled Dr Marcos to make a generous salary and maintain a lavish lifestyle.

UPMC set out to hire a surgeon who could restore the program to its former glory. It settled on Dr. Marcos, a dashing Venezuelan with a taste for Ferraris and Porsches, who specialized in the emerging field of transplants from living donors.

Furthermore,

UPMC offered Dr. Marcos $500,000 a year and "additional incentive payments," a letter dated June 21, 2002, shows. Dr. Marcos came aboard as director of clinical transplantation....

This was in a context of a medical center run by businesspeople instead of clinicians, who were increasingly richly rewarded, a context that should be familiar to Health Care Renewal readers.

UPMC is a nonprofit hospital system whose income is largely exempt from taxes. Yet, it is increasingly run like a for-profit company, paying its executives high salaries, jumping into new activities and expanding abroad. Its quest to ramp up its transplant business shows how a drive for higher revenue, now common at nonprofit hospitals, could risk compromising patient care.

Also,

Dr. Marcos's nearly six years at UPMC coincided with rapid growth at the medical center. UPMC is one of the nation's most financially successful nonprofit hospital systems, with operations ranging from Pennsylvania to Ireland and Qatar. Even though three-quarters of its $7 billion in annual revenue is exempt from federal and local taxes, UPMC has acquired many of the trappings of large, for-profit corporations.

Its chief executive, Jeffrey Romoff, earned $4 million in the fiscal year ended June 30, 2007, and 13 other employees earned in the roughly $1 million to $2 million range. For their transportation, UPMC leases a corporate jet. Earlier this year, UPMC relocated its headquarters into Pittsburgh's tallest skyscraper, the 62-story U.S. Steel Tower.

Interval Summary

So, in my humble opinion, the UPMC liver transplant story illustrates how in the US we richly reward aggressive, high-technology, cutting edge and risky care, even in the absence of any good evidence that such care provides benefits to patients that outweigh its risks. Thus, is it any surprise that we provide a lot of expensive, risky care, but may have little to show for it?

Furthermore, breaking up the medical "guild" and handing control of health care over to bureaucrats, managers, and executives paved the way for a business culture of health care that richly rewards those on top, and enables cults of personality and imperial CEOs. This has lead to a culture that puts the self-interest of leaders over the mission to care for patients. The leader-centric culture also has proved intolerant of any criticism leveled at the fearless leaders. This leads me to my last point.

A Failed Attempt to Manage the Message

We blogged earlier about how the leaders of health care organizations try to "manage the message" so as to glorify themselves and hide their own flaws. In the coverage of the UPMC liver transplant service there was plenty of attempted message managing.

First, note that no one in the UPMC leadership who bore responsibility for the transplant program was willing to say anything. Dr Marcos appeared to be in an undisclosed location.

Dr. Marcos, 46 years old when he left UPMC, did not respond to numerous attempts to reach him, including a letter sent to his home. A lawyer who represented him in a court case last year said he hadn't been in contact with Dr. Marcos for months.

Justifications of the lavish compensation given to the CEOs of health care and other large organizations often includes their heavy responsibilities. Yet the CEO of UPMC was also unavailable for comment.

UPMC declined to make Mr. Romoff available for an interview.

Instead, it was left to Mr Wood to defend the UPMC program, not only in the WSJ article, but now in several letters to the editor. Unfortunately for him, he had little ammunition for this cause.

One example was how he had to respond to the issue of whether UPMC performed transplants on patients whose liver problems were not severe enough to make them the most appropriate transplant candidates. Note that the WSJ article stated that 35% of transplant recipients during the time of Dr Marcos' leadership had MELD scores less than 15, indicating that they had favorable prognoses (without transplant.) Mr Wood responded in the WSJ article,

it wasn't until 2006 that the transplant community coalesced around a score of 15 as a cutoff to allocate organs. 'It would be unrealistic to expect a physician to practice according to yet-to-be-discovered criteria,' he said.

Actually, prioritizing liver transplants according to the recipients' MELD scores became the policy of the Organ Procurement and Transplant Network (OPTN) of the United Network for Organ Sharing (UNOS) in 2002. An article documenting the network's experience during the first year of this policy showed that the proportion of patients with mean MELD scores less than 15 ranged from about 5% to 25% across the 11 OPTN regions, with an average of about 15%.(3) This suggested there was a clear consensus to minimize the number of transplants for patients in this most favorable prognosis group by 2002. Sorry, Mr Wood.

Furthermore, yesterday Mr Wood published a letter in the Wall Street Journal in response to its article. In it, he did not provide any specific justification for the aggressive transplant policies that was based on evidence of benefits from these policies. At best, he could say,

As the pioneer and acknowledged leader in transplantation surgery, UPMC is continually exploring and testing new life-saving procedures. The use of expanded criteria organs is an attempt to alleviate a critical nationwide shortage. This alone was a compelling reason for developing a potentially groundbreaking program in living-donor liver transplant, but clearly there are risks involved whenever new procedures are developed.

This completely begged the question of whether the benefits were sufficient to offset the risks.

Furthermore, he noted,

To suggest that medical decisions are made on the basis of revenues and profits is simply ludicrous. Transplantation procedures account for less than 2% of UPMC's $7 billion in annual revenues, a fact the reporter mostly chose to ignore.

One would think an organization boasting "50,000 employees ... [and which] comprises 20 tertiary, specialty, and community hospitals, 400 outpatient sites and doctors’ offices" would pay attention to a mere 200 patients a year which provide 2% of its revenue.

He had to end up with this bit of puffery,

The real story at UPMC -- one that would clearly be of interest and importance to your readers -- is how a small, regional psychiatric hospital in Western Pennsylvania transformed itself into a self-sustaining global health enterprise that provides the highest quality medical care to patients around the world.

The advent of the internet age makes it much easier to see through attempts by health care leaders to "manage the message" (in this case, and usually by proxy) to insulate them from criticism for their actions. Maybe in the near future we can dream of health care in which incentives are made proportional to effort, ability, and most importantly the ratio of benefits to harms provided to patients.

ADDENDUM (26 November, 2008) - We were first alerted by issues with the UPMC liver transplant program in a comment on this Health Care Renewal post.

References

1. Taioli E, Marsh W. Epidemiological study of survival after liver transplant from a living donor. Transpl Int 2008: 21(10):942-7.

2. Middleton PF, Duffield M, Lynch SV, Padbury RT, House T, Stanton P, et al. Living donor liver transplantation--adult donor outcomes: a systematic review. Liver Transpl 2006; 12(1):24-30.

3. Freeman RB, Wiesner RH, Edwards E, Harper A, Merion R, Wolfe R et al. Results of the first year of the new liver allocation plan. Liver Transpl 2004; 10: 7-15.

Another Misuse of EBM

We agree with Roy Poses' comment on the recent JAMA article about EBM, particularly regarding some of its unfortunate "misuses." In the article that stands as the defining credo of Health Care Renewal, Roy wrote:

Most physicians believe that medicine should be based on science. Evidence based medicine, (EBM) ‘‘the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients’’ [74], is a refinement of this view. Meanwhile, the scientific basis of medicine is under increasingly severe attack.

We'd like to take this opportunity to identify a misuse of EBM that, ironically, contributes to the attack on the scientific basis of medicine, and that authors Montori and Guyatt didn't consider--probably because they are unaware of it. We explain it in the following letter, submitted to JAMA in response to their article. We discuss it at more length on the other blog to which we contribute, Science-Based Medicine, referenced from the letter:

To the editor:

In their recent article, “Progress in Evidence-Based Medicine,” Drs. Montori and Guyatt include important caveats about misuses of EBM. They mention trials funded by industry. They list several, documented problems with such trials, and correctly warn that these can result in erroneous interpretations by “unsophisticated users of the medical literature.” They correctly worry about “abandoning appropriate skepticism.”

There is a related misuse of EBM that leads to erroneous interpretations even by sophisticated authors, and that calls attention to systematic problems in EBM: trials of highly implausible hypotheses. Thus, regarding the homeopathic preparation Oscillococcinum, the Cochrane Collaboration writes:

Though promising, the data were not strong enough to make a general recommendation to use Oscillococcinum for first-line treatment of influenza and influenza-like syndromes. Further research is warranted…[1]

The data, however, are not promising, and further research is not warranted. The reason is that for highly implausible treatments, existing knowledge easily trumps modestly “positive” trial data. In the case of homeopathy, each of its tenets has been more than adequately disproved, and its claim that the potency of a “remedy” increases with each dilution of the original substance, continuing well past the point at which not a molecule of that substance remains in the preparation, contradicts physical reality.[2]

Such considerations reveal at least two, related problems with the evidence of EBM as it pertains to clinical trials. The first is explicit: by relegating “physiology, bench research or ‘first principles’” to Level 5 of its “Levels of Evidence” hierarchy, EBM fails to acknowledge that consistency with firmly established knowledge is a necessary, even if insufficient, basis for accepting a hypothetical treatment as effective.[2]

The second problem is part of the culture, if not the letter of EBM: by eschewing Bayesian in favor of “frequentist” statistics, its practitioners can excuse themselves from justifying estimates of prior probability, and erroneously conclude that trial data tell us more about reality than they do.[3-5]

In January, 2008, several colleagues began the blog Science-Based Medicine, referenced below.* The name, which could just as well have been “knowledge-based medicine,” is a frank recognition of the EBM weaknesses described above. Because of our interests, Science-Based Medicine’s content is predominantly, though not entirely, concerned with implausible health claims. Its larger message, however, is that these claims call attention to fundamental ways in which EBM can be improved, such as to justify its own claim to being an arbiter of real evidence.

Kimball C. Atwood, IV

Wallace I. Sampson

10/19/2008

* Steven Novella, David Gorski, Harriet Hall, Wallace Sampson, Kimball Atwood

[1] Vickers A, Smith C. Homoeopathic Oscillococcinum for preventing and treating influenza and influenza-like syndromes. Cochrane Database of Systematic Reviews 2006, Issue 3. Art. No.: CD001957. DOI: 10.1002/14651858.CD001957.pub3.

[2] Atwood K. Homeopathy and Evidence-Based Medicine: Back to the Future Part V. Science-Based Medicine 2008 (Feb. 8). http://www.sciencebasedmedicine.org/?p=42. Accessed October 19, 2008

[3] Atwood K. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine.” Science-Based Medicine 2008 (Feb. 15). http://www.sciencebasedmedicine.org/?p=48. Accessed October 19, 2008

[4] Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999;130:995-1004.

[5] Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999;130:1005-13.

"Misuses of EBM"

In an excellent article(1) just published in JAMA on progress in evidence-based medicine, Montori and Guyatt devote a section to the misuses of EBM, worth quoting in its entirety herewith emphasis added.

An analogy can be made between EBM and nuclear fission: it can be very powerful when used appropriately and dangerous when used inappropriately. The term evidence-based precedes many recommendations, guidelines, and algorithms that are not transparently linked to the underlying evidence base and do not represent the results of a systematic and critical appraisal of that evidence. It sometimes appears as if using the term obviates the need to describe the quality of underlying evidence, the magnitude of effects, or the applicability of any of the results in the context, values, and preferences of the patients.

This is particularly problematic because the EBM era has coincided with a dramatic increase in the for-profit funding of research. Researchers funded by industry interpret their results differently and in favor of the industry product relative to not-for-profit funding. Problems associated with industry funding include use of inappropriate control interventions, surrogate outcomes, publication and reporting bias, and misleading descriptions and presentations of research findings—all forms of corrupting the evidence base. Unsophisticated users of the medical literature, assuming that medical editors, peer reviewers, and topic experts have now become familiar with the tenets of EBM, may trust these corrupted research reports and advocate for their application in practice.

Many medical schools and training programs, in a form of premature closure, are moving away from teaching the fundamentals of careful evidence appraisal to emphasize the implementation of evidence. The intent of this new focus is to produce high-quality, safe, and low-cost care (ie, Accreditation Council for Graduate Medical Education competencies of systems-based practice and improvement and practice-based learning14). However, abandoning appropriate skepticism regarding the effectiveness of these interventions may lead to large investments in quality-improvement, safety, and efficiency activities that fail to yield the expected benefits.

We are truly making progress when there is now such public recognition of threats to evidence-based medicine, and thus to the rational and compassionate practice of medicine based clinical knowledge, and on both critical review of the best available clinical evidence, and sensitivity to patients' values.

I should underline the points made in the last paragraph quoted. It is distressing that what little progress in making evidence-based medicine part of medical education we made in the 1980s and 1990s is being wiped out. But what should we expect of medical schools staffed by faculty and faculty leaders who mostly are paid by the companies that make the drugs and devices that are the subject of their teaching and research?(2-3) And what should we expect of medical schools that now depend on large infusions of money from these same companies? Truly rigorous EBM might actually show that some of these "innovative" drugs and devices are not quite as good as the marketing says. So teaching same may distress the faculty's employers and the university's benefactors.

If we expect the teaching of rational and compassionate medicine that puts the benefit of patients first, we need medical schools and medical faculty who are not paid by companies whose first priority is increasing profits by selling as many drug doses and devices they can.

References

1. Montori VM, Guyatt GH. Progress in evidence-based medicine. JAMA 2008; 300: 1814-1816. Link here.

2. Campbell EG, Gruen RL, Mountford J et al. A national survey of physician–industry relationships. N Engl J Med 2007; 356:1742-1750. Link here.

3. Campbell EG, Weissman JS, Ehringhaus S et al. Institutional academic-industry relationships. JAMA 2007; 298: 1779-1786. Link here.

A Wide Web of Healthcare Data: He Who Controls The Data, Controls the Playing Field

In posts such as here and here, it's become apparent that "Evidence Based Medicine" (EBM), while perhaps reasonable in theory, is probably unreasonable in practice with the ethics of healthcare in 2008.

One cannot have evidence based rules for anything, let alone healthcare, if the evidence is tainted.

Think clinical research and the drivers of "publish or perish" are the only domains where this evidentiary taint can occur?

Think again.

From a medical colleague, a talented ER physician:

Did anyone see that Wolters Kluwer (a leading multidomestic multimedia company with corporate office in Amsterdam) plans to buy electronic medical platform Up-To-Date?

Correct me if I am off track on the issue of the "full-circle web" of information exchange - an exchange that starts with the sanctity of the private physician patient relationship.

OK, I have a private relationship with my patient, but the perverse laws say that our relationship and the information ‘collected’ is no longer sacred. It may be used without consent under the guise of 'Treatment, Payment or Healthcare Operations' (TPO).

From the Miller School of Medicine Privacy & Data Protection project glossary:

HIPAA bundles a large number of functions into the term "health care operations." This expansive list is important for many reasons, most notably because HIPAA requires no permission from patients for uses and disclosures of protected health information (PHI) for "treatment, payment or health care operations (TPO)."

Covered entities may obtain a consent for TPO-related uses and disclosures, but the practice is optional under HIPAA. (It may nonetheless be required by state law.)

Additionally, the prescription information ‘exception’ from that sensitive relationship is fair game for the data-miners to sell to Pharmacy Benefit Managers (PBMs) and the like.

The so-called ‘Publishing’ company, Wolters Kluwer, who by the way unashamedly writes friend of the court briefs supporting the data merchants and miners (e.g., Amicus Briefs supporting the plaintiffs; IMS and Verispan), is now buying one of the most widely used point of care "Evidence based medicine" tools, Up-to-date.  This is a software application for physicians to practice ‘evidence’ base medicine.

Verispan is a subdiary of one of the largest EHR vendors, McKesson. Without sounding like a nut-case full of crazy conspiracies ...

Is not this bizarre web not only destroying the trust between the patient and the physician, but additionally eroding the trust of what actually is ‘evidence’ based medicine?

I thought the rigor of science was supposed to be the driver and best influence of my decision making process for my patients, not the pull of ‘Pharma and Friends’ puppet stings.

I guess as a physician I am just marginalized to an assembly-line worker for big Pharma.

As an aside:

1. the direct to patient(consumer) marketing expenditures of Pharma – est. $30 billion / year.
2. the entire NIH budget $28 bil/yr.

So, it seems, evidence based medicine is not only threatened by commercial taint of clinical trials and research, it's also potentially threatened by the drivers and behaviors of the Data Merchant component of the healthcare IT ecosystem.

Who polices them, I ask?

-- SS