Delfini Logo
Evidence- and Value-based Solutions for Health Care™
Clinical Improvement Consults, Content Development, Training & Seminars, Tools

Publications: Performance Measures

The EBM Information Quest: Is it true? Is it useful? Is it usable?™

Delfini Co-founders: Michael E Stuart MD, President & Medical Director . Sheri Ann Strite, Managing Director & Principal

Quick Picks

Delfini: Dr. Michael E. Stuart & Sheri Ann Strite
Why Critical Appraisal Matters



Delfini Group Publishing

Contact Us
Updates & Contact Info

Free Online Tools

Free Online Tutorial

Delfini Blog

EBM Dolphin
Evidence & Quality Improvement Commentaries


Follow & Share...

Just-in-time UpdatesFollow Delfini Group on Twitter

Like Us Like Us on Facebook  Find UsFind Us at LinkedIn

Modern Healthcare Performance Measurement Issue

Special Article:
Modern Healthcare: Pitfalls to Performance Measurement

• Stuart M, Strite SA. 2005 Jan 30. Pitfalls to Performance Measurement. Modern Healthcare. Originally published 2006 Jan 23. Accessed 2006 Jan 30.

About Modern Healthcare


DelfiniGram™: GET ON OUR UPDATE LIST Contact Us


Performance measurement can be an important strategy to help improve clinical care. And performance measurement is an area that is fraught with pitfalls which have the potential to undermine quality, penalize quality performers and reward others who are not providing quality care. We offer practical help for how to develop good performance measures.

Delfini Peformance Measurement Resources

Performance Measurement Training

Sample Topics

  • Introduction to Evidence-based Medicine & the Evidence-based Organization
  • Introduction to Performance Measurement
  • Validity Essentials: Randomization, Bias, Confounding & Chance
  • Performance Measurement Essentials
  • Finding Seed Performance Measures
  • Pay-for-Performance Essentials
  • Avoiding Pitfalls & Directing Attention to Quality: Or How Not to "Get Had"
  • Using Evidence, Guidelines and Performance Measures to Improve Care
  • Sample Program Agenda (see November 6th)

Performance Measure Training Testimonials

Modern Healthcare: Pitfalls to Performance Measurement

• Stuart M, Strite SA. 2005 Jan 30. Pitfalls to Performance Measurement. Modern Healthcare. Originally published 2006 Jan 23. Accessed 2006 Jan 30.

Mike Stuart in Modern HealthcareWeb Exclusive: Written by Michael Stuart and Sheri Ann Strite and reprinted here with permission.

Pitfalls to performance measurement

When drawing conclusions from data, watch out for conflicting measures

Story originally published January 23, 2006

Regulators, payers, patients and others understandably want to determine which healthcare providers deliver the highest quality of care. Interest in performance measurement has increased since such groups such as the Institute of Medicine, the Center for the Evaluative Clinical Sciences at Dartmouth, the CMS, and the Rand Corp. have reported estimates that 20% to 50% of all prescriptions, visits, procedures and hospitalizations in the U.S. may be inappropriate. That translates into patient deaths and injury and the waste of an estimated $400 billion annually.

Clearly there is a need to improve healthcare, and performance measurement is an increasingly applied component of quality improvement. Yet there are pitfalls to performance measurement of which many users are unaware.

We offer an outline of some important limitations we believe healthcare leaders and managers should consider as they attempt to improve quality through the use of performance measures within their organizations. Providers must distinguish between three main uses of performance measures: required reporting, quality improvement, and evaluating organizational or individual performance. Each of these uses requires a different approach because of the possibility of invalid measures. Providers must understand the limitations and pitfalls of drawing conclusions from performance measurement because of the problems resulting from conflicting measures, which can result in decreased quality, waste and incorrect conclusions about individual health organizations or practitioners.

Components of a measure
A good performance measure consists of a valid numerator and denominator, frequency of occurrence and the data-gathering process. It is often expressed as a percentage or a rate. The text statement for a diabetic patient-care performance measure might be, "The percent of patients with a diagnosis of diabetes mellitus receiving at least one hemoglobin A1c annually."

A valid denominator means that it specifies the right base from which the measurement will be made. The denominator must have appropriate criteria for the pool of whom or what is eligible for measurement. For example, a denominator measuring clinical improvement for cervical Pap smears that does not exclude women without a cervix would be invalid.

Numerators generally count events such as something that happens to a patient or something patients receive - typically, this is an outcome, an intervention, a service or a process. Examples of numerators are the number of patients with diabetes mellitus from the denominator (the population eligible for measurement) who have received an annual hemoglobin A1c, or the percent of charts available for patient appointments at an outpatient clinic.

A key point -- often misunderstood -- is that numerators should be based on valid, useful and usable scientific evidence. For health status outcomes and interventions, this requires that we know what interventions are likely to result in improvement. With rare exceptions, only well-designed and conducted randomized controlled trials can demonstrate cause and effect relationships, and only valid and useful information from trials data should be used for interventions.

Medical leaders, administrators and others should be aware of the significant potential for selection bias, observation bias, confounding and the play of chance when relying on observational data -- and performance measures are one form of observational data. These threats to validity can confound performance comparisons between institutions, units and individuals. Trying to resolve this by adjusting for case mix is analogous to using models to adjust for patient differences in observational studies dealing with therapy -- the potential to be misled by confounding factors remains high. Databases and clinical records are useful for measuring processes, but are not reliable for attempting to "prove" that a health status improvement was the result of an intervention. Database information, observational studies and opinions of experts can inadvertently mislead.

Unfortunately, because of a general lack of knowledge of these potential problems, providers may be required to report outcomes to stakeholders even when the performance measure or the organization's outcomes lack validity.

Problematic results
Feedback to a clinician in the form of a performance report may be of great value as a way of encouraging his or her participation in quality improvement efforts and focusing attention on improving processes of care and attention to patients' needs. However, because of significant validity and reliability problems inherent in observational data, it is altogether a different issue when an individual provider's performance is made available to others in the form of a performance "report card" or when an individual's income is based on a limited set of performance measures.

Here are several examples of problems that can result:

  • The wrong denominator: A colon cancer screening quality improvement project at the Veterans Administration Hospital in San Francisco resulted in the facility failing to meet a national target and the hospital faced financial penalties. However, an audit revealed that 47% of the patients included in the measure had declined screening, 12% failed to make their appointments for screening, 11% had chart documentation that screening was not indicated and 42% of the counted patients received diagnostic testing rather than screening (i.e., they had signs or symptoms of disease). Thus, the conclusion that the hospital was failing to meet national VA benchmarks was incorrect.

  • The wrong numerator: Some groups recommend routine screening of all newborns for hearing problems during postpartum hospitalization-this is even required by law in many states. There is, however, insufficient evidence to conclude that such testing leads to improved speech and language skills at 3 years of age. It is also unclear from the best available evidence if potential benefits outweigh potential harms of false-positive tests. Unfortunately, many stakeholders demand performance data on this measure. When looking at internal quality projects it would be better to select a more valid measure.

  • Problems in judging the quality of a clinician: A physician may take appropriate actions to improve quality of care, but because of patient factors, systems factors or small sample size, the physician's performance may not result in clinical improvement.

Examining the use of profiling family physicians for glycemic control in their diabetic patients is instructive. It has been reported that in a typical family practice, only 4% or less of variance in hospitalization rates, visit rates, lab utilization and glycemic control in diabetics can be attributed to differences in physician practice patterns. For profiles of glycemic control, outlier physicians could dramatically improve their profiles by pruning their panels of as few as one to three patients with the highest HbA1cs levels. This gaming of the system could not be prevented by case-mix adjustment.

Leaders and managers should remember that conclusions about individuals -- and even organizations -- based on performance measures alone should be drawn with great care. A key issue is how much of the outcome is due to selection bias, sample size, and other factors such as how much of the outcome is really under the control of the clinician or the health system and how much is not.

Performance measurement is an important component of quality improvement efforts in healthcare. However, if performance measures are not designed and used correctly, they are rendered statistically or clinically meaningless -- a waste of resources and a threat to quality. To bring clinicians on board with quality efforts and to improve patient safety and outcomes, we must ensure that each measure captures the data intended and is used appropriately.

Michael Stuart is president of the Delfini Group and clinical assistant professor in the Department of Family Medicine, University of Washington School of Medicine, Seattle. Sheri Ann Strite is a principal and managing partner in the Delfini Group, Portland, Ore.


Contact UsCONTACT DELFINI Delfini Group EBM DolphinDelfini Group EBM Dolphin

At Publications: PM

EBM Delfini boat

Read Our Blog...

Use of our website implies agreement to our Notices. Citations for references available upon request.

Best of Delfini
What's New


Delfini Group Publishing
Sample Projects
About Us & Our Work
Site Search

Contact Info/Updates


Quick Navigator to Selected Resources


Return to Top

© 2002-2020 Delfini Group, LLC. All Rights Reserved Worldwide.
Use of this website implies your agreement to our Notices.

EBM Solutions for Evidence-based Health Care Quality