Delfini Logo
Evidence- and Value-based Solutions for Health Care™
Clinical Improvement Consults, Content Development, Training & Seminars, Tools

Critical Appraisal Matters

The EBM Information Quest: Is it true? Is it useful? Is it usable?™

Delfini Co-founders: Michael E Stuart MD, President & Medical Director . Sheri Ann Strite, Managing Director & Principal

Quick Picks

Delfini: Dr. Michael E. Stuart & Sheri Ann Strite
Why Critical Appraisal Matters

Services
Services

Seminars
Seminars

Delfini Group Publishing
Books

Contact Us
Updates & Contact Info

Tools
Free Online Tools

Tutorial
Free Online Tutorial

blog
Delfini Blog


EBM Dolphin
Delfini
Click
Evidence & Quality Improvement Commentaries

 

Follow & Share...

Just-in-time UpdatesFollow Delfini Group on Twitter

Like Us Like Us on Facebook  Find UsFind Us at LinkedIn

 

DelfiniGram™: GET ON OUR UPDATE LIST Contact Us

At "Why Critical Appraisal Matters"

EBM Delfini boat

Read Our Blog...

Menu........
Use of our website implies agreement to our Notices. Citations for references available upon request.

Home
Best of Delfini
What's New
Blog

Seminars

Services
Delfini Group Publishing
Resources
Sample Projects
Notices
About Us & Our Work
Testimonials
Other
Site Search

Contact Info/Updates

........

Quick Navigator to Selected Resources

.......................


"Why Critical Appraisal Matters" Menu........

  1. Lack of Critical Appraisal May Hurt Patients
  2. More Problems
  3. Solutions
  4. Resources
  5. Videos
  6. Pre-Test Failure Rates Report
  7. Contact Delfini
  8. References
  9. Downloadable summary
  10. Delfini Flier: Do You Need Help with Overuse & Patient Satisfaction?
  11. Delfini Flier: The Medical Misinformation Mess—Problems; Solutions
  12. Delfini Flier: Ways You Need Help With Overuse & Patient Safety You Probably Didn't Even Know About
  13. Article: Down the Rabbit Hole of Medical Misinformation by Renee Despres and published on Your Medical Source

........

Lack of Critical Appraisal May Hurt Patients

Much medical science is not reliable or of uncertain reliability. Worse, health care providers frequently do not know how to detect unreliable medical science. Health care providers frequently do not know how to interpret study results—even basic ones. Roughly, over 70 percent of physicians, clinical pharmacists and other health care professionals fail our simple 3-question critical appraisal training program pre-test [Delfini].

To give you a flavor of how serious this failure is, one question simply assesses the person's awareness of a need for a comparison group in an efficacy study. Without a comparison group, how would you know a condition wasn't self-limiting, for example, and recovery not due to an intervention?

Understanding basic requirements for valid medical science is very important. Health care providers owe it to patients to acquire these skills. Otherwise, patients can be harmed or miss opportunities for true consent.

Numbers below are estimates, but are likely to be close or understated. To put these numbers into context, roughly 58,000 US lives were lost in the Vietnam war. [ref last accessed 10/08/2012]

  • Over 63,000 people with heart disease died after taking encainide or flecainide because many doctors thought taking these drugs "made biological sense," but did not understand the simple need for reliable clinical trial information to confirm what seemed to "make sense" [Echt 91].

  • An estimated 60,000 people in the United States died and another 140,000 experienced a heart attack resulting from the use of a nonsteroidal anti-inflammatory drug despite important benefit and safety information reported in a key research trial published in a prestigious medical journal [Graham].

  • In another example, roughly 42,000 women with advanced breast cancer suffered excruciating side effects without any proof of benefit, many of them dying as a result, and at a cost of $3.4 billion dollars [Mello].

  • At least 64 deaths out of 751 cases in nearly half the United States were linked to fungal meningitis thought to be caused by a contaminated treatment that is used for back and radicular pain—but there is no reliable scientific evidence of benefit from that treatment [CDC].

In the above instances, these were preventable deaths and harms—from common treatments—which patients might have avoided if their physicians had better understood the importance and methods of evaluating medical science.

........

More Problems

Most of us know that there is much variation in healthcare that is not explained by patient preference, differences in disease incidence or resource availability. However did you know—

Fact 1. Healthcare in the United States is shockingly poor. [Chassin, CMS 10, Kerr, McGlynn, Skinner]

Leading experts estimate that 20 to 50 percent of all healthcare in the United States is inappropriate. This leads to patient harms and waste (at least 100 billion dollars annually). Here are some big reasons why we think there is so much inappropriate care...

Fact 2. Training in medical schools and other schools for allied health professionals in the United States is shockingly poor when it comes to training in science.

Training of healthcare professionals is generally lacking in understanding medical science. Douglas Altman wrote of this problem many years ago in an editorial titled "The scandal of poor medical research" [Altman] which was recently remarked on by Richard Smith in a BMJ blog entited, "Medical research—still a scandal." [Smith]

As a result, the majority of physicians, clinical pharmacists and others working in health care medical decision-making, (and this includes academicians, researchers, editors and peer reviewers) do not know how to conduct a reliable study nor how to evaluate whether a research study is valid and the results, clinically useful. Most who believe they have these skills do not. Read our Report on Critical Appraisal Skills Failure Rates.

Fact 3. This affects the quality of medical research and the quality of medical care.

We have long estimated that less than 10 percent of all medical research—regardless of source—is reliable or clinical useful. Others agree. Professor John Ioannidis "...charges that as much as 90 percent of the published medical information that doctors rely on is flawed.” We think this is just the tip of the iceberg. Read Delfini Letter and Commentary on "Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010. [Strite, Freedman] In one survey of 60,352 studies, a meager 7% passed criteria of high quality methods and clinical relevancy [McKibbon], and fewer than 5% passed a validity screening for an evidence-based journal. [Glasziou]

Fact 4. There is nowhere to turn for trustable healthcare information.

FDA approval is not sufficient for establishing scientific validity and usefulness.

We know of no fully "trustable" healthcare information sources, and sources that claim to be "evidence-based" frequently are not. Some of the best and "most trusted" sources have frequently failed our critical appraisal audits.

Most secondary sources are based on invalid studies or studies that do not have clinically meaningful outcomes. This includes reviews, meta-analyses, performance measures, clinical recommendations, pharmacoeconomic studies, disease management protocols and more. Clinical guidelines vary in quality and the majority may be invalid, including many from professional societies.

Fact 5. Possessing skills in critical appraisal matters.

Bias in studies tends to favor the intervention under investigation. Certain kinds of bias have been shown to distort research results up to a relative 50 percent or more—for each flaw. [References for the evidence on the distorting effects of are available upon request to delfini (at) delfini.org.]

Furthermore, there are a lot of other problems that possessing critical appraisal skills are likely to solve—here are just a few examples...

Most physicians rely on abstracts which are frequently inaccurate. One study found that 18-68 percent of abstracts in 6 top-tier medical journals contained information not verifiable in the body of the article.[Pitkin] One study concluded that there may be considerable bias in p-values reported in abstracts.[Gøtzsche] Physicians and others who understand critical appraisal know it cannot be determined whether a study is valid by reading the abstract.

Many physicians rely on information that should be treated as hypothesis-generating only. Physicians who understand critical appraisal know how to avoid being mislead by this.

Author's conclusions are opinions, not evidence—and authors are often biased, even with the best of intentions. Authors frequently use misleading terms or draw misleading conclusions. Physicians and others who lack critical appraisal skills often get "had" by this, whereas critical appraisal knowledge can frequently defend against these problems.

Physicians and others who do not understand issues with findings that are not statistically significant frequently mistakenly interpret these findings as meaning there is no meaningful difference between the groups. Those with critical appraisal skills understand how to use confidence intervals to avoid these erroneous interpretations.

........

Solutions

Attaining critical appraisal skills need not be hard or time-consuming—individuals can easily acquire the basic skills and without learning much about statistics. Skills can be easily acquired in a short period of time.

Healthcare providers owe it to their patients to gain these skills. There are many resources available—both formal and informal—including self-teaching modules available online. We provide a wealth of materials online for self-study for free and have several books available for purchase.

We also offer fee-based services and can teach people basic skills for analyzing studies of interventions in a short period of time.

Our programs are not boring, nor are they too hard. We utilize a practical, simplified and applied tool-based approach in an engaging and entertaining way.

“...thanks for presenting this valuable information in a fun and memorable way...”

"...I thought the subject was going to be boring and not much use...I was very wrong...I cannot thank you and Sheri enough."

More Testimonials.

........

Resources

What are the core competencies required for critical appraisal for studies of interventions? Download our 1-page critical appraisal checklist and find out: [WORD] & [PDF]

Here are some resources for new learners:

1-Pager: Basics of Evaluating Evidence in Superiority Trials for Therapies [PDF]

Short critical appraisal tutorial: EVALUATING BIASED EFFICACY RESULTS IN SUPERIORITY TRIALS

Additional Delfini tools to help with evidence-based quality improvement can be found at the Delfini Tools & Educational Library.

 

Delfini Group Publishing

Basics For Evaluating Medical Research Studies:
A Simplified Approach

And Why Your Patients Need You To Know This

Delfini Group Evidence-based Practice Series
Short How-to Guide Book

...................................................

"Best help with evidence-based medicine available."
Martin Gabica, MD, Chief Medical Officer, Healthwise
...................................................

This book is about how to evaluate the reliability and clinical usefulness of clinical trials. Written for physicians and other health care professionals, this book is written in easy-to-understand terms that even the layperson can understand and put to use. Now available for purchase.

See all Delfini books available including for P&T committees, medical technology assessments, evidence-based clinical improvements and even for patients.

........

Speciality Videos: The Medical Misinformation Mess & Why Critical Appraisal Matters

1. "Medical Misinformation Mess" Solutions— Communicating Reliable & Useful Evidence Must be at the Heart of Clinical Quality Improvement for Good Outcomes, Value & Informed Consent 11 min

Video Thumb

Much published medical science is of uncertain reliability. This can hurt patients through missed opportunities for right care, risk for harms and waste and insufficient information to choose decisions most right for them. Here are our recommendations for change to give patients the care that is their right and that they deserve.

2. Critical Appraisal Matters—Stories: COX2s—The Power of Reliable and Useful Evidence to Improve Outcomes, Value & Informed Consent 20 min

Video Thumbnail

Much published medical science is of uncertain reliability. This can hurt patients through missed opportunities for right care, risk for harms and waste and insufficient information to choose decisions most right for them. Here are a few stories of why critical appraisal of the medical literature is needed to provide patients with reliable information to make good decisions.

3. Delfini Book: About "What You Don't Know Can Hurt You—A Guide for Patients" 9 min

Video Thumbnail

Many times patients accept medical advice without understanding they have a choice. Many times patients are given medical advice that is not supported by reliable science. In many cases, these situations have resulted in harms to patients they might have avoided if they understood some little known facts about the health care world and communicated with their health care providers differently.

This book is about the need for you to sometimes change the course of your care based on the medical information you receive—or have not received. This book is about how to talk to your doctors and other care providers to improve your opportunities for getting the information you need to make health care choices that are best for you.

For a compelling journey about this book, go to the book's website where you will also find our video...

We also have an online tutorial and videos on critical appraisal here.

........

Contact UsCONTACT DELFINI Delfini Group EBM DolphinDelfini Group EBM Dolphin

........

References

  1. Altman DG. The scandal of poor medical research. BMJ. 1994 Jan 29;308(6924):283-4. PubMed PMID: 8124111; PubMed Central PMCID: PMC2539276.
  2. Centers for Medicare and Medicaid Services, Office of the Actuary, National Health Statistics Group, National Health Care Expenditures Data, January 2010.
  3. CDC:  http://www.cdc.gov/hai/outbreaks/meningitis.html
  4. Chassin MR, Galvin RW, the National Roundtable on Health Care Quality. The urgent need to improve health care quality: Institute of Medicine National Roundtable on Health Care Quality [consensus statement]. JAMA. 1998;280:1000–1005.
  5. Cohen SP. Epidural steroid injections for low back pain. BMJ. 2011 Sep 13;343:d5310. doi: 10.1136/bmj.d5310. PubMed PMID: 21914757.
  6. Delfini: http://www.delfini.org/Delfini_Pre-Test_Report_0306.pdf
  7. Echt DS, Liebson PR, Mitchell LB, Peters RW, Obias-Manno D, Barker AH, Arensberg D, Baker A, Friedman L, Greene HL, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med. 1991 Mar 21;324(12):781-8. PubMed PMID: 1900101.
  8. Freedman DH. Lies, Damned Lies, and Medical Science. The Atlantic. November 2010.
  9. Glasziou P. The EBM journal selection process: how to find the 1 in 400 valid and highly relevant new research articles. Evid Based Med. 2006 Aug;11(4):101. PubMed PMID: 17213115.
  10. Gøtzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 2006;333;231-234; PMID: 16854948
  11. Graham: http://www.naturalnews.com/011401_Dr_David_Graham_the_FDA.html
  12. Iversen T, Solberg TK, Romner B, et al. Effect of caudal epidural steroid or saline injection in chronic lumbar radiculopathy: multicentre, blinded, randomised controlled trial. BMJ. 2011 Sep 13;343:d5278. doi: 10.1136/bmj.d5278. PubMed PMID: 21914755.
  13. Kerr EA, McGlynn EA, Adams J, et al. Profiling the quality of care in twelve communities: Results from the CQI study. Health Aff. 2004;23:247–256.
  14. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645.
  15. McKibbon KA, Wilczynski NL, Haynes RB. What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med. 2004 Sep 6;2:33. PubMed PMID: 15350200; PubMed Central PMCID: PMC518974.
  16. Mello MM, Brennan TA. The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer. Health Aff (Millwood). 2001 Sep-Oct;20(5):101-17. PubMed PMID: 11558695.
  17. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of Data in Abstracts of Published Research Articles. JAMA. 1999; 281: 1110-1111. [PMID 10188662]
  18. References for the distorting effects of bias are available upon request to delfini (at) delfini.org.
  19. Skinner J, Fisher ES, Wennberg JE; for the National Bureau of Economic Research. The efficiency of Medicare. Working Paper No. 8395. Cambridge, MA: National Bureau of Economic Research; July 2001.
  20. Smith, R. Medical research—still a scandal. 01/31/2014. http://blogs.bmj.com/bmj/2014/01/31/richard-smith-medical-research-still-a-scandal/
  21. Strite SA, Stuart ME. Letter to the Editor: How to Evaluate Medical Science. The Atlantic. January/February 2011.

 

Return to Top

© 2002-2017 Delfini Group, LLC. All Rights Reserved Worldwide.
Use of this website implies your agreement to our Notices.

EBM Solutions for Evidence-based Health Care Quality