Study highlights 'lurking question' of measuring EHR effectiveness: The science in Medical Informatics is dead

The science in Medical Informatics is dead.

I'm not going to even use academic fabric softener in my assertion, e.g., "may be", "appears to be", or "is it?" (as a question) dead.

It's dead.

When HIT experts recommend changing the study goalposts when existing studies don't give results they'd like to see, rather than first and foremost critically and rigorously examining why we're seeing unexpected results, science is dead.

http://www.healthcareitnews.com/news/study-highlights-lurking-question-measuring-ehr-effectiveness

Study highlights 'lurking question' of measuring EHR effectiveness

December 22, 2010 | Molly Merrill, Associate Editor

WASHINGTON – Hospitals' use of electronic health records has had just a limited effect on improving the quality of medical care nationwide, according to a study by the nonprofit RAND Corporation.

The study, published online by the American Journal of Managed Care, is part of a growing body of evidence suggesting that new methods should be developed to measure the impact of health information technology on the quality of hospital care.

[In other words, we're not getting the results we thought and hoped we'd get with "Clinical IT 1.0", so let's alter the study methodologies and endpoints --- rather than using the results we have to identify the causes and improve the technology to see if we can do better with "Clinical IT 2.0."

Further, it's not as if there's no other data on why health IT might not work as hoped - ed.]

Most of the current knowledge about the relationship between health IT and quality comes from a few hospitals that may not be representative, such as large teaching hospitals or medical centers that were among the first to adopt electronic health records.

[This implies "other" "representative" hospitals are either not doing it right, or the technology is ill suited for them and may never work. Which is it? We really need to know before we proceed with hundreds of billions more in this "Grand Experiment" - ed.]

The RAND study is one of the first to look at a broad set of hospitals to examine the impact that adopting electronic health records has had on the quality of care.

The research included 2,021 hospitals – about half the non-federal acute care hospitals nationally. Researchers determined whether each hospital had EHRs and then examined performance across 17 measures of quality for three common illnesses – heart failure, heart attack and pneumonia. The period studied spanned from 2003 to 2007.

The number of hospitals using either a basic or advanced electronic health records rose sharply during the period, from 24 percent in 2003 to nearly 38 percent in 2006.

[How many billions of dollars diverted from patient care needs does that represent? - ed.]

Researchers found that the quality of care provided for the three illnesses generally improved among all types of hospitals studied from 2004 to 2007. The largest increase in quality was seen among patients treated for heart failure at hospitals that maintained basic electronic health records throughout the study period.

However, quality scores improved no faster at hospitals that had newly adopted a basic electronic health record than in hospitals that did not adopt the technology.

[In other words, the improvements or lack thereof had little to do with electronic vs. paper record keeping - ed.]

In addition, at hospitals with newly adopted advanced electronic health records, quality scores for heart attack and heart failure improved significantly less than at hospitals that did not have electronic health records.

[In other words, the clinical IT was probably impairing doctors compared to simpler paper methods and good HIM personnel - ed.]

EHRs had no impact on the quality of care for patients treated for pneumonia.

Researchers say the mixed results may be attributable to the complex nature of healthcare.

[That is likely true, but maybe the mixed results are also -- and even more likely in major part -- due to poorly designed and/or poorly implemented IT - ed.]

Focusing attention on adopting EHRs may divert staff from focusing on other quality improvement efforts.

[That speaks to poor EHR overkill, poor usability, unfitness for purpose, and other issues that may or may not be remediable in short or even long term - ed.]

In addition, performance on existing hospital quality measures may be reaching a ceiling where further improvements in quality are unlikely.

[That speaks to a low ROI or even negative for the hundreds of billions of dollars being diverted to the IT sector - ed.]

"The lurking question has been whether we are examining the right measures to truly test the effectiveness of health information technology," said Spencer S. Jones, the study's lead author and an information scientist at RAND. "Our existing tools are probably not the ones we need going forward to adequately track the nation's investment in health information technology."

["Probably" not the ones we need? How can the authors know this? This is not science, it is speculation. Further, I'd say the scientific imperative before we design "the right measures" to "truly" test the effectiveness of HIT is to understand why the measures we're using now are not showing the desired results, because perhaps they are perfectly adequate and are revealing crucial flaws, overestimations and false assumptions that need to be dealt with, now, not after another round of billions is spent - ed.]

New performance measures that focus on areas where EHRs are expected to improve care should be developed and tested, according to researchers.

[In pharma clinical trials, this is akin to what is known as "changing the study methodologies and endpoints" - a form of manipulating clinical research, usually with the true ultimate endpoint of money - ed.]

For example, EHRs are expected to lower the risk of adverse drug interactions, but existing quality measures do not examine the issue.

[I believe the studies that have been done of CPOE have not been consistently supportive, and in fact show CPOE might create new med errors - ed.]

"With the federal government making such a large investment in this technology, we need to develop a new set of quality measures that can be used to establish the impact of electronic health records on quality," Jones said.

[This is truly putting the cart before the horse as I wrote here. The studies showing the benefit should have long preceded the "large investments" that were decided upon - ed.]

Support for the study was provided by RAND COMPARE (Comprehensive Assessment of Reform Efforts). RAND developed COMPARE to provide objective facts and analysis to inform the dialogue about health policy options. COMPARE is funded by a consortium of individuals, corporations, corporate foundations, private foundations and health system stakeholders.

Other authors of the study are John L. Adams, Eric C. Schneider, Jeanne S. Ringel and Elizabeth A. McGlynn.

The overarching assumption is that the metrics are wrong, not the quality and fitness for purpose of the technology, the 'wrongness' of which is painfully obvious from the aforementioned other literature, e.g., link, link, link, and the many posts at this blog referring to other literature. Are the authors unaware, one might ask? I know they are not. (Or - blinded? That is, could there be external pressures affecting the thought processes? The arguments might not unreasonably be construed to be skewed from that perspective.)

Recognizing the atrocious user experience and mission hostile nature of the technology (link), how disruptive it is (link, link), how poorly implemented it is often by domain amateurs to support financial battles of the payers, not cognitive processes of clinicians, I am amazed there's any signs of improvement at all, not outright deterioration. (That there is not outright deterioration of care displays if anything the results of the hard mental labor and ingenuity of clinicians to work around the technology's deficits.)

I note that the last time RAND looked at such matters there were problems with pro-health IT bias, among other issues (see my Feb. 2009 post "Heartland Institute Research & Commentary: Health Information Technology").

Carl Sagan wrote that science is a candle in the dark in a demon haunted world.

It seems the demons are winning.

-- SS

Hospitals Under the Knife: Sacrificing Hospital Jobs for the Extravagance of Healthcare IT?

A WSJ article on the financial condition of NY hospitals, and specifically a line by NY Mayor Bloomberg, caught my eye:

Wall Street Journal
Mar. 26, 2010
Hospitals Under the Knife
New York City System Aims to Cut 2,600 More Jobs as State Funding Drops
By MICHAEL HOWARD SAUL and SUZANNE SATALINE

NEW YORK—The nation's largest public hospital system plans to slash its work force—including doctors and nurses—by about 10% over two years as government aid drops and the number of uninsured patients jumps.

With its budget deficit set to top $1 billion, New York City's Health and Hospitals Corp. plans to eliminate 2,600 jobs in the fiscal year that begins July 1. That comes on top of 1,300 positions to be eliminated this year."No hospital system in the country is exempt from the crushing economics facing the health-care industry," said New York Mayor Michael Bloomberg. He noted that New York had been early to adopt electronic medical records but said that state budget cuts were hitting the system hard.

... Previous job cuts focused on trimming support staff, but the new measures will include physicians and nurses, Alan Aviles, the corporation's president, said in an interview.

"Early to adopt EMR's BUT the budget cuts were hitting the system hard?"

To the knowledgeable, this seems a non sequitur. Its message is clearly that yes, we spent hundreds of millions of dollars on EMR's, but the adoption of EMR's should have saved us jillions of dollars, helping insulate us from economic downturns.

Yet some very serious researchers say this is not the case.

For starters, there's, Ashish Jha’s research at the Harvard School of Public Health that compared 3,000 hospitals at various stages in the adoption of computerized health records and found little difference in the cost and quality of care. A New York Times story "Little Benefit Seen, So Far, in Electronic Patient Records" on those findings is here. Was anyone in the Governor's office or hospital governance reading their own newspaper?

Then, there's the Nov. 2009 “Hospital Computing and the Costs and Quality of Care: A National Study” (Amer J Med 123:1; 40-46) by Himmelstein and Wololhandler at Harvard Medical School, that also concluded “as currently implemented, hospital computing might [very] modestly improve process measures of quality but not administrative or overall costs."

There's the June 2009 Wharton School of Business article "Information Technology: Not a Cure for the High Cost of Health Care" that I wrote about at this HC Renewal post. Senior Wharton professors wrote:

Technology could increase health care costs without markedly improving quality, according to experts at Wharton.

... "No one has done the careful research to indicate that if one health care system has information technology and the other doesn't, then the care is different. There are no controlled trials," says Mark Pauly, a health care management professor at Wharton. All that technology is no panacea, he warns. In fact, he believes IT could actually raise costs because of culture clashes, training, the implementation of the systems [I would say "the mayhem that often goes on during the implementation" - ed.] and the labor required to maintain the new technology.

"The best-case scenario is that information technology will improve quality but not lower costs. The worst case is that there's no difference at all."

... That opinion is echoed by other experts at Wharton and the University of Pennsylvania. "The focus on IT in health care is a good thing, but there's way too much hype about it and misunderstanding about what the benefits will be and how quickly they will come," says Peter Gabriel, medical director of clinical information systems at the University of Pennsylvania Health System.

[Kevin Volpp, professor of medicine and health care management] agrees that tracking real cost savings from health care IT is a difficult task, but he expects there to be some benefits from spotting and eliminating redundant care. But those benefits aren't likely to add up to big savings, says Lawton R. Burns, director of the Wharton Center for Health Management and Economics. "I agree that information technology is important, but it's not the slam dunk it's portrayed to be," he says. The chase to reduce costs, improve quality and expand coverage is deemed the "iron triangle of health care. A lot of us wince [at that goal]," Burns notes. "It's arguable that we can't do any of those things well."

David A. Asch, a Wharton health care management and economics professor, agrees that technology is a big part of reform. "No one is arguing against it, but that doesn't mean that it's not oversold," he says. Gabriel likens the fascination over IT in health care to a shiny new object that's easier to focus on relative to more daunting issues.

... In addition, it's unclear what cultural issues [a big theme in my writings - ed.] will emerge as information technology is adopted. These cultural issues are in the forefront of primary care physician relationships. Experts at Wharton and Penn say physicians are generally skeptical of the technology movement. How much will a technology overhaul add to operating costs? How much will it cost to retrain workers? What's the electronic record learning curve? And what happens when a doctor has a laptop between him and the patient?

"Individual physicians just don't know where the money is going to come from," says Pauly. "If IT is tied to reimbursements it could work, but [many] are skeptical." Burns adds that the physician-patient relationship can also be altered. "Technology adoption changes the way you practice. What happens when your primary care physician is looking at his screen instead of you?"

There are the concerns of Abraham Verghese, Professor and Senior Associate Chair for the Theory and Practice of Medicine at Stanford, who wrote in the Wall Street Journal in a June 2009 article "The Myth of Prevention" that:

... I have similar problems with the way President Obama hopes to pay for the huge and costly health reform package he has in mind that will cover all Americans; he is counting on the “savings” that will come as a result of investing in preventive care and investing in the electronic medical record among other things. It’s a dangerous and probably an incorrect projection.

Finally, this is an experimental technology whose benefits and risks are not well known.

In the article "Electronic Health Record Use and the Quality of Ambulatory Care in the United States" (Arch Intern Med. 2007;167:1400-1405), the authors examined electronic medical records use throughout the U.S. and the association of EMR use with 17 basic quality indicators. They concluded that “as implemented, EMRs were not associated with better quality ambulatory care.”

Further, the FDA recently testified that this technology can actually harm and kill patients, but the extent is unknown. Existing FDA data is likely the "tip of the iceberg", testified FDA official Jeffrey Shuren MD, JD at the HIT Policy Committee, Adoption/Certification Workgroup, special meeting on health IT safety on February 25, 2010.

I described Shuren's testimony in my HC Renewal post "FDA on Health IT Adverse Consequences: 44 Reported Injuries And 6 Deaths, Probably Just 'Tip of Iceberg'." Considering FDA is nearly unknown as the go-to for such reports, and the low reporting of medical errors related to computing noted by Koppel at the same meeting, the actual rates of injury and death could be much, much larger.

In effect, NY hospital physicians, nurses and support staff will lose their job due to budget shortfalls, at the same time the NY hospitals have been spending hundreds of millions of dollars on the extravagance of experimental clinical IT systems whose benefit is still an unknown.

Perhaps some of those millions could have been better spent on human beings, such as employees or better yet, patient care.

As I've written before, the health IT industry seems to be staging an invasion of healthcare to its own benefit. Now, clinical personnel are losing their jobs as a result, and patient care is likely to suffer.

While I'm supportive of EMR experimentation when finances are stable (when performed with patient safety considerations as paramount, of course), this is not the time for such extravagance in NY, especially when jobs - both support and clinical - are threatened.

Irrational exuberance in technology is bad enough - it's far worse when you can't afford the objects of your affection.

One suggestion is that the healthcare IT experiments be put on hold as unaffordable under current conditions, and resumed when finances are more stable. The money could be diverted to keeping physicians, nurses and support staff employed. The risks could also be studied further. However, this might cause some executives somewhere to have to forgo their pet contracts with their friendly HIT vendors and management consultant companies.

Computers are more important than people, after all.

At a time of massive international economic difficulty, "Blood for Computers!" can be the new rallying cry.

Since many of the layoffs will involve union members of District Council 37, the city's largest municipal employee union, perhaps the rallying cry "Computers for Union Busting!" could also apply.

(Some people would have no problems with that, but these are real, live hospital staff being put out of work, and the patients they care for being affected.)

-- SS

3/26/10 Addendum:

The advice above may apply to an entire country, the UK, that seems to have spent about £13 billion (about $19 billion U.S.) on health IT that doesn't work.

Per the Telegraph article today "Hospital wards to shut in secret NHS cuts":

Tens of thousands of NHS workers would be sacked, hospital units closed and patients denied treatments under secret plans for £20 billion of health cuts.

... The proposals could lead to:

• 10 per cent of NHS staff being sacked in some areas.
• The loss of thousands of hospital beds.
• A reduction in the number of ambulance call-outs.
• Medical professionals being replaced by less qualified assistants.

At least the IT industry is alive and well.

-- SS