Critical Appraisal Tutorial
The EBM Information Quest: Is it true? Is it useful? Is it usable?™
Follow & Share...
At Critical Appraisal Tutorial
Quick Navigator to Selected Resources
Key Reading: "How To Survive the Medical Misinformation Mess" by John P. A. Ioannidis, Michael E. Stuart, Shannon Brownlee, Sheri A. Strite, published in the European Journal of Clinical Investigation and, within fewer than four months after publication, one of the top 20 papers in 2017 downloaded from The European Journal of Clinical Investigation.
An obvious question raised by the title is, “Get you where?” Well, the answer is, “To where you know it is reasonable to think you can trust the results of the study you have just finished reading.” In this blog, our focus is on how to critically appraise medical research studies which claim superiority for efficacy of a therapy.
Because of Lack of Understanding Medical Science Basics, People May Be Injured or Die
Understanding basic requirements for valid medical science is very important. Numbers below are estimates, but are likely to be close or understated—
In the above instances, these were preventable deaths and harms—from common treatments—which patients might have avoided if their physicians had better understood the importance and methods of evaluating medical science.
Failures to Understand Medical Science Basics
Many health care professionals don’t know how to quickly assess a trial for reliability and clinical usefulness—and yet mastering the basics is not difficult. Over the years, we have given a pre-test of 3 simple questions to more than a thousand physicians, pharmacists and others who have attended our training programs. Approximately 70% fail—"failure" being defined as missing 2 or 3 of the questions.
One pre-test question is designed to see if people recognize the lack of a comparison group in a report of the "effectiveness" of a new treatment. Without a comparison group of people with similar prognostic characteristics who are treated exactly the same except for the intervention under study, you cannot discern cause and effect of an intervention because a difference between groups may explain or affect the results.
A second pre-test question deals with presenting results as relative risk reduction (RRR) without absolute risk reduction (ARR) or event rates in the study groups. A "relative" measure raises the question, "Relative to what?" Is the reported RRR in our test question 60 percent of 100 percent? Or 60 percent of 1 percent?
The last of our pre-test questions assesses attendees’ basic understanding of only one of the two requirements to qualify as an Intention-to-Treat (ITT) analysis. The two requirements are that people should be randomized as analyzed and that all people should be included in the analysis whether they have discontinued, are missing or have crossed over to other treatment arms. The failure rate at knowing this last requirement is very high. (We will add that this last requirement means that a value has to be assigned if one is missing—and so, one of the most important aspects of critically appraising an ITT analysis is the evaluation of the methods for "imputing" missing data.)
By the end of our training programs, success rates have always markedly improved. Others have reported similar findings.
There is a Lot of Science + Much of It May Not Be Reliable
Bias Distorts Results
Patients Deserve Better
Some Simple Steps To Critical Appraisal
You can review and download a wealth of freely available information at our website at www.delfini.org including checklists and tools at http://www.delfini.org/delfiniTools.htm which can provide you with much greater information. Most relevant to this blog is our short critical appraisal checklist which you can download here—http://www.delfini.org/Delfini_Tool_StudyValidity_Short.pdf
Also, you can view a video (very high level!) here and down load the accompanying slides here. http://www.delfini.org/page_Good_EBM_Tips.htm#videos
The Big Questions
There are four stages of a clinical trial, and you should ask several key questions when evaluating bias in each of the 4 stages.
Important considerations include how were subjects selected for study, were there enough subjects, how were they assigned to their study groups, and were the groups balanced in terms of prognostic variables.
Your critical appraisal to-do list includes—
a) Checking to see if the randomization sequence was generated in an acceptable manner. (Minimization may be an acceptable alternative.)
b) Determining if the investigators adequately concealed the allocation of subjects to each study group? Meaning, is the method for assigning treatment hidden so that an investigator cannot manipulate the assignment of a subject to a selected study group?
c) Examining the table of baseline characteristics to determine whether randomization was likely to have been successful, i.e., that the groups are balanced in terms of important prognostic variables (e.g., clinical and demographic variables).
What is being studied, and what is it being compared to? Was the intervention likely to have been executed successfully? Was blinding likely to have been successful? Was duration reasonable for treatment as well as for follow-up? Was adherence reasonable? What else happened to study subjects in the course of the study such as use of co-interventions? Were there any differences in how subjects in the groups were treated?
Your to-do list includes evaluating—
a) Adequacy of blinding of subjects and all working with subjects and their data—including likely success of blinding;
b) Subjects’ adherence to treatment—special attention should be given to imbalance in attrition between the groups;
c) Inter-group differences in treatment or care except for the intervention(s) being studied.
What information was collected, and how was it collected? What data are missing and is it likely that missing data could meaningfully distort the study results?
Your to-do list includes evaluating—
a) Measurement methods (e.g., mechanisms, tools, instruments, means of administration, personnel issues, etc.)
b) Classification and quantification of missing data in each group (e.g., discontinuations due to ADEs, unrelated deaths, protocol violations, loss to follow-up, etc.)
c) Whether missing data are likely to distort the reported results? This is the area that the evidence on the distorting risk of bias provides the least help. And so, again, often it is worthwhile asking, "What conditions would need to be present for attrition to distort the results?"
Were outcome measures reasonable, pre-specified and analyzed appropriately? Was reporting selective? How was safety assessed? Remember that models are not truth.
Your to-do list includes evaluating—
a) Whether assessors were blinded.
b) How the effect size was calculated (e.g., absolute risk reduction, relative risk, etc.). You especially want to know benefit or risk with and without treatment.
c) Were confidence intervals included? (You can calculate these yourself online, if you wish. See our web links at our website for suggestions.)
d) For dichotomous variables, was a proper intention-to-treat (ITT) analysis conducted with a reasonable choice for imputing values for missing data?
e) For time-to-event trials, were censoring rules unbiased? Were the number of censored subjects reported?
After you have evaluated a study for bias and chance and have determined that the study is valid, the study results should be evaluated for clinical meaningfulness, (e.g., the amount of clinical benefit and the potential for harm). Clinical outcomes include morbidity; mortality; symptom relief; physical, mental and emotional functioning; and, quality of life—or any surrogate outcomes that have been demonstrated in valid studies to affect a clinical outcome.
Sample critical appraisal:
It is not difficult to learn how to critically appraise a clinical trial. Health care providers owe it to their patients to gain these skills. Health care professionals cannot rely on abstracts and authors’ conclusions—they must assess studies first for validity and second for clinical usefulness. Authors are often biased, even with the best of intentions. Remember that authors’ conclusions are opinions, not evidence. Authors frequently use misleading terms or draw misleading conclusions. Physicians and others who lack critical appraisal skills are often mislead by authors’ conclusions and summary statements. Critical appraisal knowledge is required to evaluate the validity of a study which must be done prior to seriously considering reported results.
Critical appraisal tool for Secondary Studies (i.e., studies of studies such as meta-analyses) is here.
Critical appraisal tool for Secondary Sources (i.e., anything using a primary or secondary study such as clinical guidelines, compendia, cost-effectiveness studies, etc.) is here.
For those who wish to go more deeply, we have books available and do training seminars. See our website at www.delfini.org.
Attaining critical appraisal skills need not be hard or time-consuming—individuals can easily acquire the basic skills and without learning much about statistics. Skills can be easily acquired in a short period of time.
Healthcare providers owe it to their patients to gain these skills. There are many resources available—both formal and informal—including self-teaching modules available online. We provide a wealth of materials online for self-study for free and have several books available for purchase.
We also offer fee-based services and can teach people basic skills for analyzing studies of interventions in a short period of time.
Our programs are not boring, nor are they too hard. We utilize a practical, simplified and applied tool-based approach in an engaging and entertaining way.
“...thanks for presenting this valuable information in a fun and memorable way...”
"...I thought the subject was going to be boring and not much use...I was very wrong...I cannot thank you and Sheri enough."
Here are some resources for new learners:
1-Pager: Basics of Evaluating Evidence in Superiority Trials for Therapies [PDF]
Free video tutorials.
"How to Critically Appraise A Medical Research Study: Fast . Efficient . Effective"
Download Slide Printout
Downloads for Tools Used in the Video
The Professional Tool
Embedded Demo Only—"How to Read a Medical Research Study: The Quick Mark-up Method"