"Why Critical Appraisal Matters" Menu........
- Lack of Critical Appraisal May Hurt Patients
- More Problems
- Pre-Test Failure Rates Report
- Contact Delfini
- Downloadable summary
- Delfini Flier: Do You Need Help with Overuse & Patient Satisfaction?
- Delfini Flier: The Medical Misinformation Mess—Problems; Solutions
- Delfini Flier: Ways You Need Help With Overuse & Patient Safety You Probably Didn't Even Know About
- Article: Down the Rabbit Hole of Medical Misinformation by Renee Despres and published on Your Medical Source
Lack of Critical Appraisal May Hurt Patients
Much medical science is not reliable or of uncertain reliability. Worse, health care providers frequently do not know how to detect unreliable medical science. Health care providers frequently do not know how to interpret study results—even basic ones. Roughly, over 70 percent of physicians, clinical pharmacists and other health care professionals fail our simple 3-question critical appraisal training program pre-test [Delfini].
To give you a flavor of how serious this failure is, one question simply assesses the person's awareness of a need for a comparison group in an efficacy study. Without a comparison group, how would you know a condition wasn't self-limiting, for example, and recovery not due to an intervention?
Understanding basic requirements for valid medical science is very important. Health care providers owe it to patients to acquire these skills. Otherwise, patients can be harmed or miss opportunities for true consent.
Numbers below are estimates, but are likely to be close or understated. To put these numbers into context, roughly 58,000 US lives were lost in the Vietnam war.
[ref last accessed 10/08/2012]
Over 63,000 people with heart disease died after taking encainide or flecainide because many doctors thought taking these drugs "made biological sense," but did not understand the simple need for reliable clinical trial information to confirm what seemed to "make sense" [Echt 91].
An estimated 60,000 people in the United States died and another 140,000 experienced a heart attack resulting from the use of a nonsteroidal anti-inflammatory drug despite important benefit and safety information reported in a key research trial published in a prestigious medical journal [Graham].
In another example, roughly 42,000 women with advanced breast cancer suffered excruciating side effects without any proof of benefit, many of them dying as a result, and at a cost of $3.4 billion dollars [Mello].
At least 64 deaths out of 751 cases in nearly half the United States were linked to fungal meningitis thought to be caused by a contaminated treatment that is used for back and radicular pain—but there is no reliable scientific evidence of benefit from that treatment [CDC].
In the above instances, these were preventable deaths and harms—from common treatments—which patients might have avoided if their physicians had better understood the importance and methods of evaluating medical science.
of us know
that there is much variation in healthcare that is not explained by
patient preference, differences in disease incidence or resource availability.
However did you know—
1. Healthcare in the United States is shockingly poor. [Chassin, CMS 10, Kerr, McGlynn, Skinner]
experts estimate that 20 to 50 percent of all healthcare
in the United States is inappropriate. This leads to
patient harms and waste (at least 100 billion dollars annually).
Here are some big reasons why we think there is so much inappropriate
Training in medical schools and other schools for allied health professionals
in the United States is shockingly poor when it comes to training
Training of healthcare
professionals is generally lacking in understanding medical
science. Douglas Altman wrote of this problem many years ago in an editorial titled "The scandal of poor medical research" [Altman] which was recently remarked on by Richard Smith in a BMJ blog entited, "Medical research—still a scandal." [Smith]
As a result, the majority of physicians, clinical
pharmacists and others working in health care medical decision-making,
(and this includes academicians, researchers, editors and peer reviewers)
do not know how to conduct a reliable study nor how to evaluate whether
a research study is valid and the results, clinically useful. Most
who believe they have these skills do not. Read our Report
Appraisal Skills Failure Rates.
This affects the quality of medical research and the quality of medical
We have long estimated
that less than 10 percent of all medical research—regardless
of source—is reliable or clinical useful. Others agree.
Professor John Ioannidis "...charges that as much as 90 percent
of the published medical information that doctors rely on is flawed.”
We think this is just the tip of the iceberg. Read Delfini
Letter and Commentary on "Lies, Damned Lies, and Medical Science,"
by David H. Freedman, The Atlantic, November 2010. [Strite, Freedman] In one survey of 60,352 studies, a meager 7% passed criteria of high quality methods and clinical relevancy [McKibbon], and fewer than 5% passed a validity screening for an evidence-based journal. [Glasziou]
There is nowhere to turn for trustable healthcare information.
is not sufficient for establishing scientific validity and
We know of no
fully "trustable" healthcare information sources,
and sources that claim to be "evidence-based" frequently
are not. Some of the best and "most trusted" sources
have frequently failed our critical appraisal audits.
sources are based on invalid studies or studies that do
not have clinically meaningful outcomes. This includes
reviews, meta-analyses, performance measures, clinical recommendations,
pharmacoeconomic studies, disease management protocols and more.
Clinical guidelines vary in quality and the majority may be invalid, including many from professional societies.
5. Possessing skills in critical appraisal matters.
in studies tends to favor the intervention under investigation.
Certain kinds of bias have been shown to distort research
results up to a relative 50 percent or more—for
each flaw. [References for the evidence on the distorting effects of are available upon request to delfini
there are a lot of other problems that possessing critical appraisal
skills are likely to solve—here are just a few examples...
rely on abstracts which are frequently inaccurate. One study found
that 18-68 percent of abstracts in 6 top-tier medical journals contained
information not verifiable in the body of the article.[Pitkin]
One study concluded that there may be considerable bias in p-values
reported in abstracts.[Gøtzsche] Physicians and others who understand critical
appraisal know it cannot be determined whether a study is valid
by reading the abstract.
rely on information that should be treated as hypothesis-generating
only. Physicians who understand critical appraisal know
how to avoid being mislead by this.
are opinions, not evidence—and authors are often biased, even
with the best of intentions. Authors frequently use misleading
terms or draw misleading conclusions. Physicians and others
who lack critical appraisal skills often get "had" by
this, whereas critical appraisal knowledge can frequently defend
against these problems.
others who do not understand issues with findings that are not
statistically significant frequently mistakenly interpret
these findings as meaning there is no meaningful difference between
the groups. Those with critical appraisal skills understand how
to use confidence intervals to avoid these erroneous interpretations.
Attaining critical appraisal skills need not be hard or time-consuming—individuals
can easily acquire the basic skills and without learning much about
statistics. Skills can be easily acquired in a short period of time.
Healthcare providers owe it to their patients to gain these skills.
There are many resources available—both formal and informal—including
self-teaching modules available online. We provide a wealth of materials online for self-study for free and have several books available for purchase.
We also offer
fee-based services and can teach people basic skills for analyzing
studies of interventions in a short period of time.
programs are not boring, nor are they too hard. We utilize a practical,
simplified and applied tool-based approach in an engaging and entertaining
“...thanks for presenting this valuable information in a fun and memorable way...”
thought the subject was going to be boring and not much use...I was
very wrong...I cannot thank you and Sheri enough."
are the core competencies required for critical appraisal for studies
of interventions? Download our 1-page critical appraisal checklist and
find out: [WORD]
Here are some resources for new learners:
1-Pager: Basics of Evaluating Evidence in Superiority Trials for Therapies [PDF]
Short critical appraisal tutorial: EVALUATING BIASED EFFICACY RESULTS IN SUPERIORITY TRIALS
Additional Delfini tools to help with evidence-based quality improvement can be found
at the Delfini Tools & Educational Library.
Basics For Evaluating Medical Research Studies:
A Simplified Approach
And Why Your Patients Need You To Know This
Delfini Group Evidence-based Practice Series
Short How-to Guide Book
"Best help with evidence-based medicine available."
Martin Gabica, MD, Chief Medical Officer, Healthwise
This book is about how to evaluate the reliability and clinical usefulness of clinical trials. Written for physicians and other health care professionals, this book is written in easy-to-understand terms that even the layperson can understand and put to use. Now available for purchase.
See all Delfini books available including for P&T committees, medical technology assessments, evidence-based clinical improvements and even for patients.
Speciality Videos: The Medical Misinformation Mess & Why Critical Appraisal Matters
1. "Medical Misinformation Mess" Solutions— Communicating Reliable & Useful Evidence Must be at the Heart of Clinical Quality Improvement for Good Outcomes, Value & Informed Consent 11 min
Much published medical science is of uncertain reliability. This can hurt patients through missed opportunities for right care, risk for harms and waste and insufficient information to choose decisions most right for them. Here are our recommendations for change to give patients the care that is their right and that they deserve.
2. Critical Appraisal Matters—Stories: COX2s—The Power of Reliable and Useful Evidence to Improve Outcomes, Value & Informed Consent 20 min
Much published medical science is of uncertain reliability. This can hurt patients through missed opportunities for right care, risk for harms and waste and insufficient information to choose decisions most right for them. Here are a few stories of why critical appraisal of the medical literature is needed to provide patients with reliable information to make good decisions.
3. Delfini Book:
"What You Don't Know Can Hurt You—A Guide for Patients" 9 min
Many times patients accept medical advice without understanding they have a choice. Many times patients are given medical advice that is not supported by reliable science. In many cases, these situations have resulted in harms to patients they might have avoided if they understood some little known facts about the health care world and communicated with their health care providers differently.
This book is about the need for you to sometimes change the course of your care based on the medical information you receive—or have not received. This book is about how to talk to your doctors and other care providers to improve your opportunities for getting the information you need to make health care choices that are best for you.
For a compelling journey about this book, go to the book's website where you will also find our video...
We also have an online tutorial and videos on critical appraisal here.
- Altman DG. The scandal of poor medical research. BMJ. 1994 Jan 29;308(6924):283-4. PubMed PMID: 8124111; PubMed Central PMCID: PMC2539276.
- Centers for Medicare and Medicaid Services, Office of the Actuary, National Health Statistics Group, National Health Care Expenditures Data, January 2010.
- CDC: http://www.cdc.gov/hai/outbreaks/meningitis.html
- Chassin MR, Galvin RW, the National Roundtable on Health Care Quality. The urgent need to improve health care quality: Institute of Medicine National Roundtable on Health Care Quality [consensus statement]. JAMA. 1998;280:1000–1005.
- Cohen SP. Epidural steroid injections for low back pain. BMJ. 2011 Sep 13;343:d5310. doi: 10.1136/bmj.d5310. PubMed PMID: 21914757.
- Delfini: http://www.delfini.org/Delfini_Pre-Test_Report_0306.pdf
- Echt DS, Liebson PR, Mitchell LB, Peters RW, Obias-Manno D, Barker AH, Arensberg D, Baker A, Friedman L, Greene HL, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med. 1991 Mar 21;324(12):781-8. PubMed PMID: 1900101.
- Freedman DH. Lies, Damned Lies, and Medical Science. The Atlantic. November 2010.
- Glasziou P. The EBM journal selection process: how to find the 1 in 400 valid and highly relevant new research articles. Evid Based Med. 2006 Aug;11(4):101. PubMed PMID: 17213115.
- Gøtzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 2006;333;231-234; PMID: 16854948
- Graham: http://www.naturalnews.com/011401_Dr_David_Graham_the_FDA.html
- Iversen T, Solberg TK, Romner B, et al. Effect of caudal epidural steroid or saline injection in chronic lumbar radiculopathy: multicentre, blinded, randomised controlled trial. BMJ. 2011 Sep 13;343:d5278. doi: 10.1136/bmj.d5278. PubMed PMID: 21914755.
- Kerr EA, McGlynn EA, Adams J, et al. Profiling the quality of care in twelve communities: Results from the CQI study. Health Aff. 2004;23:247–256.
- McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645.
- McKibbon KA, Wilczynski NL, Haynes RB. What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med. 2004 Sep 6;2:33. PubMed PMID: 15350200; PubMed Central PMCID: PMC518974.
- Mello MM, Brennan TA. The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer. Health Aff (Millwood). 2001 Sep-Oct;20(5):101-17. PubMed PMID: 11558695.
- Pitkin RM, Branagan MA, Burmeister LF. Accuracy of Data in Abstracts of Published Research Articles. JAMA. 1999; 281: 1110-1111. [PMID 10188662]
- References for the distorting effects of bias are available upon request to delfini (at) delfini.org.
- Skinner J, Fisher ES, Wennberg JE; for the National Bureau of Economic Research. The efficiency of Medicare. Working Paper No. 8395. Cambridge, MA: National Bureau of Economic Research; July 2001.
- Smith, R. Medical research—still a scandal. 01/31/2014. http://blogs.bmj.com/bmj/2014/01/31/richard-smith-medical-research-still-a-scandal/
- Strite SA, Stuart ME. Letter to the Editor: How to Evaluate Medical Science. The Atlantic. January/February 2011.