Delfini Logo
Evidence- and Value-based Solutions for Health Care™
Clinical Improvement Consults, Content Development, Training & Seminars, Tools

Recommended Reading

The EBM Information Quest: Is it true? Is it useful? Is it usable?™

Delfini Co-founders: Michael E Stuart MD, President & Medical Director . Sheri Ann Strite, Managing Director & Principal

Quick Picks

Delfini: Dr. Michael E. Stuart & Sheri Ann Strite
Why Critical Appraisal Matters



Delfini Group Publishing

Contact Us
Updates & Contact Info

Free Online Tools

Free Online Tutorial

Delfini Blog

EBM Dolphin
Evidence & Quality Improvement Commentaries


Follow & Share...

Just-in-time UpdatesFollow Delfini Group on Twitter

Like Us Like Us on Facebook  Find UsFind Us at LinkedIn


Text BookDelfini Group PublishingDelfini Evidence-based Practice Short How-to Guide Book now available for purchase.





DelfiniGram™: GET ON OUR UPDATE LIST Contact Us

This is generally a list of readings from sources other than Delfini. We include our work where especially relevant; but otherwise look for other relevant readings from our work using the Quick Navigator.


At Recommended Reading

Recommended Reading

Read Our Blog...

Use of our website implies agreement to our Notices. Citations for references available upon request.

Best of Delfini
What's New


Delfini Group Publishing
Sample Projects
About Us & Our Work
Site Search

Contact Info/Updates


Quick Navigator to Selected Resources


Clinical Quality Improvement

Strite S, Stuart M
The EBM Organization
: Three-part series by Delfini published in The Physician Executive, Journal of Medical Management

Rosser WW, Davis D, Gilbart E.
Assessing Guidelines for Use in Family Practice.
Journal of Family Practice
2001; 50: 969-973.

Hayward RSA et al.
User’s Guides to the Medical Literature, VIII: How to Use Clinical Practice Guidelines, A: Are the recommendations valid?
. 1995; 274: 570-574.

Wilson MC et al.
User’s Guides to the Medical Literature, VIII: How to Use Clinical Practice Guidelines, B: What are the recommendations and will they help you in caring for your patients?
. 1995; 274: 1630-1632.

Bero LA et al. for the Cochrane Effective Practice and Organization of Care Review Group.
Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings.

NHS Centre for Review and Dissemination.
Effective Health Care. Getting Evidence into Practice.
London, England: Royal Society of Medicine Press;1999.

Oxman AD et al.
No magic bullets: a systematic review of 102 trials of interventions to improve professional practice.

Also see Delfini Health Care System Tool Set.

Formulary Management

Starred FavoritesStrite SA, Stuart ME, Urban S.
Process steps and suggestions for creating drug monographs and drug class reviews in an evidence-based formulary system.
Formulary. April 2008;43:135–145.

Strite SA, Stuart ME.
An Introduction to Systematic Reviews
: The Lifeblood of Medical Decision-making.
California Pharmacist 2008. Vol LV, No. 3. Summer 2008: 52-56.

Chiquette E, Posey LM.
Evidence-based Pharmacotherapy.
American Pharmacists Association. APhA Publications; 1 edition (January 15, 2007).
Contributing Authors Include
Strite SA, Stuart ME, contributing authors. Chapter 7. Applying Evidence-Based Pharmacotherapy to Formulary Decisions

Here is survey work that describes needs and perceptions:

Ryan Rodriguez, PharmD, BCPS; Brett J. Kelly, BSPharm, MS; and Mary Moody, BSPharm
Evaluating the Training, Responsibilities, and Practices of
P&T Committee Members and Nonmember Contributors

Here is a paper that presents a perspective from a group we worked with:

Raulo S. Frear, PharmD
Experimental and Observational Data and Formulary Listing

Balance Sheets

Braddick M, Stuart M, Hrachovec J.
The use of balance sheets in developing clinical guidelines.
J Am Board Fam Pract. 1999 Jan-Feb;12(1):48-54. Erratum in: J Am Board Fam Pract 1999 Mar-Apr;12(2):187. PMID: 10050643

Made available to our readers through the kind permission of the Journal of the American Board of Family Medicine. Their webside is here: See above link to read the article.


Glasziou P, Haynes B. The paths from research to improved health outcomes. Evid Based Nurs. 2005 Apr;8(2):36-8. PubMed PMID: 15830412.

Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, Vale L. Toward evidence-based quality improvement. Evidence (and its limitations) of he effectiveness of guideline dissemination and implementation strategies 1966-1998. J Gen Intern Med. 2006 Feb;21 Suppl 2:S14-20.

Grol R et al.
Evidence-based Implementation of Evidence-based Medicine
Journal of Quality Improvement. 1999; 25: 10; 503-513.
PubMed Abstract:

Grol R et al.
Attributes of clinical guidelines that influence use of guidelines in general practice: observational study.
BMJ 1998; 317:858-861.
Full Text:

Lomas J, Haynes RB.
A taxonomy and critical review of tested strategies for the application of clinical practice recommendations from “official” to “individual” clinical policy.
AM J Prevent Med. 1988;4:77-94
PubMed Reference:

BookRogers EM.
Diffusion of Innovations. 3rd Edition. The Free Press. New York; 1983.

Scott IA, Glasziou PP. Improving effectiveness of clinical medicine: the need for better translation of science into practice. Med J Aust. 2012 Oct 1;197(7):374-8. PubMed PMID: 23025728.

Solberg L et al.
Lessons from Experienced Guideline Implementers: Attend to Many Factors and Use Multiple Strategies.
Journal of Quality Improvement. 2000; 26: 171-188.
PubMed Abstract:

Wensing, M et al.
Implementing guidelines and innovations in general practice: which interventions are effective?
British Journal of General Practice. Feb 1998; 48:991-997.
PubMed Abstract:

Yank V, Tribett E, Green L, Pettis J, Learning from Marketing: Rapid Development of Medication Messages that Engage Patients, Patient Education and Counseling (2015),

Performance Measurement

Stuart M, Strite S
Pitfalls to Performance Measurement:
Article by Delfini published in Modern Healthcare

National Quality Measures Clearinghouse (NQMC): Includes many educational documents and an archive of quality and performance measures

Bagley B.
How does your practice measure up?
Fam Pract Manag. 2006 Jul-Aug;13(7):59-64.
PMID: 16909830

Nerenz DR, Neil N
Performance Measures for Health Care Systems
To access, copy the following into an internet search engine:

Vijan, Sandeep
Are We Overvaluing Performance Measures?
Effective Clinical Practice ACP site:

Landon BE, Normand ST, Blumenthal D, Daley J.
Physician clinical performance assessment: prospects and barriers.
JAMA. 2003;290:1183-1189.

Walter LC, Davidowitz NP, Heineken PA et al.
Pitfalls of Converting Practice Guidelines into Quality Measures: Lessons Learned from a VA Performance Measure.
JAMA. 2004;2901:2466-2470. Abstract.

Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG.
The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease.
JAMA. 1999 Jun 9;281(22):2098-105.
PMID: 10367820 Abstract.

Evidence-based Medicine

Here is a nice, short general write up on what evidence-based medicine is. NOTE: We do have a couple of issues. 1. We would amend to say "best available valid and clinically useful evidence." 2. And we are often asked about industry research. We say that everyone has a potential bias — critically appraise every study, every secondary source, for validity and clinical usefulness. And yet, we also think overall this is a nice intro to EBM.

Dickersin K, Straus SE, Bero LA.
Evidence based medicine: increasing, not dictating, choice.
BMJ. 2007 Jan 6;334 Suppl 1:s10. PMID: 17204750.

Starred Favorites Cochrane Handbook
The Cochrane Handbook for Systematic Reviews of Interventions is a useful resource for all healthcare professionals doing critical appraisal of the medical literature. It was created to assist authors preparing systematic reviews (SRs) but is of great value to all of us who are assessing validity of studies. The section of interest is Part 2: General methods for Cochrane reviews.

Starred Favorites General Books on Evidence-based Medicine Topics & Clinical Epidemiology

Eddy, David
Clinical Decision Making: From Theory to Practice: A Collection of Essays From the Journal of the American Medical Association
Jones & Bartlett Pub; 1st edition (January 15, 1996)

Fletcher, Robert H, Suzanne W. Fletcher and Edward H. Wagner
Clinical Epidemiology: The Essentials.
Philadelphia PA. Lippincott Williams & Wilkins, 1996.

Geyman, John P., Richard A. Deyo, Scott D. Ramsey
Evidence-based Clinical Practice — Concepts and Approaches
Butterworth Heinemann, 2000.

Guyatt, Gordon and Rennie Drummond, ed. The Evidence-Based Medicine Working Group
Users' Guides to the Medical Literature: Essentials of Evidence-Based Clinical Practice.
AMA Press, 2002.

Jekel, James F., Joann G. Elmore, David L. Katz
Epidemiology, Biostatistics and Preventive Medicine
WB Saunders Company, 1996

Riegelman, Richard
Studying a Study and Testing a Test.
Philadelphia PA. Lippincott Williams & Wilkins, 2000.

Straus SE, Richardson WS, Glasziou P, Haynes RB. Evidence-based Medicine: How to Practice and Teach EBM. 3rd Ed. Elsevier Churchill Livingston, 2005.

Click here to go to Delfini critical appraisal tools in our Evidence Tool Set. Click here for Delfini Primers.

Critical Appraisal — General

Delfini Primers & Evidence Tools

Starred FavoritesThe Consort Statement: Consolidated Standards of Reporting Trials —CONSORT
"The CONSORT statement is an important research tool that takes an evidence-based approach to improve the quality of reports of randomized trials. The statement is available in six languages and has been endorsed by prominent medical journals such as The Lancet, Annals of Internal Medicine, and the Journal of the American Medical Association. Its critical value to researchers, health care providers, peer reviewers, and journal editors, and health policy makers is the guarantee of integrity in the reported results of research.

CONSORT comprises a checklist and flow diagram to help improve the quality of reports of randomized controlled trials. It offers a standard way for researchers to report trials. The checklist includes items, based on evidence, that need to be addressed in the report; the flow diagram provides readers with a clear picture of the progress of all participants in the trial, from the time they are randomized until the end of their involvement. The intent is to make the experimental process more clear, flawed or not, so that users of the data can more appropriately evaluate its validity for their purposes."

Gluud LL.
Bias in Clinical Intervention Research.
Am J Epidemiol 2006;163:493–501
PMID: 17119141

Ioannidis JPA (2005)
Why most published research findings are false.
PLoS Med 2(8): e124
PMID: 16060722
Direct access here.

Juni P, Altman DG, Egger M.
Systematic reviews in health care: Assessing the quality of controlled clinical trials.
BMJ. 2001 Jul 7;323(7303):42-6.
PMID: 11440947

Schulz KF, Grimes DA.
Generation of allocation sequences in randomised trials: chance, not choice.
Lancet 2002; 359: 515–19.
PMID: 11853818
Concealment of Allocation
Viera AJ, Bangdiwala SI.
Eliminating bias in randomized controlled trials: importance of allocation concealment and masking.
Fam Med. 2007 Feb;39(2):132-8.

PMID: 17273956
Direct access here.

Boutron I, Estellat C, Guittet L, Dechartres A, Sackett DL, et al. (2006)
Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: A systematic review. PLoS Med 3(10): e425. DOI: 10.1371/ journal. PMID: 17076559
Direct access here.

Flum DR. Interpreting Surgical Trials With Subjective Outcomes Avoiding UnSPORTsmanlike Conduct. JAMA, November 22/29, 2006—Vol 296, No. 20: 2483-1484. PMID: 17119146
See Delfini commentary in related DelfiniClick™

Jüni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001 Jul 7;323(7303):42-6. Review. PubMed PMID: 11440947; PubMed Central PMCID: PMC1120670.

Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001 Dec 4;135(11):982-9. Erratum in: Ann Intern Med. 2008 Aug 5;149(3):219. PubMed PMID: 11730399.

Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998 Aug 22;352(9128):609-13. PubMed PMID: 9746022.

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of of treatment effects in controlled trials. JAMA 1995;273:408­12. PMID: 7823387
See Delfini commentary in related DelfiniClick™

Viera AJ, Bangdiwala SI. Eliminating bias in randomized controlled trials: importance of allocation concealment and masking.Fam Med. 2007 Feb;39(2):132-8. PMID: 17273956
Direct access here.

Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008 Mar 15;336(7644):601-5. Epub 2008 Mar 3. PubMed PMID: 18316340; PubMed Central PMCID:PMC2267990.
See Delfini commentary in related DelfiniClick™ on blinding and objective outcomes.

And on Testing for Success of Blinding:

Sackett DL. Turning a blind eye: why we don't test for blindness at the end of our trials. BMJ. 2004 May 8;328(7448):1136. PubMed PMID: 15130997; PubMed Central PMCID: PMC406365.
See Delfini commentary in related DelfiniClick™.

Analysis (see Intention-to-Treat Analysis also)

Kaplan Meier curves explained:

See also Delfini Tools & Educational Library for 1-pager.

Intention-to-Treat Analysis

Schulz KF, Grimes DA
Sample size slippages in randomised trials: exclusions and the lost and wayward.
The Lancet
. Vol 359. March 2, 2002: 781-785
PMID: 11888606
NOTE: Delfini stresses that the approach taken for missing values should not give an advantage to the intervention.

Hollis S, Campbell F
What is meant by intention to treat analysis? Survey of published randomised controlled trials.
. Vol 319. Sept 1999: 670-674

NOTE: Delfini agrees that differential loss is important to note, but even equivalent loss of greater than five percent could be a threat to validity.

More reading: Intention-to-Treat Analysis — The Biased Case of Migraine at our Evidence Essentials Page.

Click here to go to Delfini critical appraisal tools in our Evidence Tool Set and here for Delfini White Paper on Missing Data.

Advanced Reading
Lachin JM.
Statistical considerations in the intent-to-treat principle.
Control Clin Trials 2000;21:167–189. PMID: 11018568

Placebo Effect

Not reading exactly, but incredibly informative and entertaining:

The Strange Powers of the Placebo Effect

Assessing Harms

Starred FavoritesEffective Health Care
Methods Guide for Comparative Effectiveness Reviews
Assessing Harms When Comparing Medical Interventions

Citation: Chou R, Aronson N, Atkins D, et al. Assessing harms when comparing medical interventions. In: Agency for Healthcare Research and Quality. Methods Guide for Comparative Effectiveness Reviews [posted November 2008]. Rockville, MD.

Systematic Reviews & Meta-analyses

Strite SA, Stuart ME.
An Introduction to Systematic Reviews
: The Lifeblood of Medical Decision-making.
California Pharmacist 2008. Vol LV, No. 3. Summer 2008: 52-56.

Oxman AD, Cook DJ, Guyatt GH. How to use an Overview.
User’s Guides to Evidence-based Medicine.
JAMA. 1994 Nov 2; 272 (17): 1367-71

Egger M, Smith GD, Phillips AN
Meta-analysis: principles and procedures.
. 1997: 315:1533-1537.
(BMJ has a great site in their collections:

Miser WF
Applying a Meta-Analysis to Daily Clinical Practice.
J Am Board Fam Pract
13(3) : 201-210, 2000.
This tool is currently unavailable online.

Delfini suggestions and commentary on the Miser article and evaluation tool:

• Under “Was the method for selecting articles…” Question 3. Was selection done blindly and in random order – means “Were the articles presented in random order to reviewers and were reviewers blinded [communication from author]. All relevant studies should be reviewed.

Often it is recommended that all studies selected should go in the same direction of treatment effect. The issue is trying to ensure that the studies were similar enough to combine — if the results are very different, that may point to a dissimilarity in studies. All relevant studies should be sought using selection criteria, including those in grey literature to address potential for publication bias of lack of reporting of negative finding studies. Doing systematic reviews is complicated. Sensitivity analyses should be done and reported, along with a robust discussion by the authors of potential differences in studies.

Also, we've included on our site our own Delfini Systematic Review Validity Tool. Click here to go to Delfini critical appraisal tools in our Evidence Tool Set.

Diagnostic Testing

Jaeschke R et al.
How to Use an Article About a Diagnostic Test. Users’ Guides to Evidence-based Medicine

Greenhalgh T.
How to read a paper: Papers that report diagnostic or screening tests.
BMJ 1997; 315:540-3

BookFletcher, Robert H, Suzanne W. Fletcher and Edward H. Wagner Clinical Epidemiology: The Essentials. Philadelphia PA. Lippincott Williams & Wilkins, 1996.

Bandolier at

Delfini Diagnostic Testing Calculator Online [Excel]
• Natural language explanations for results. Can use with patients.
• Provides information in percents, probabilities and odds.
• Likelihood ratio results computed for you so you don't have to guess what is “insignificant” and what is “large.” The calculator does the interpretive work for you.
• Bandolier number-needed-to-diagnose (NND) calculated as well.
• Calculates on the basis of —
   • pre-test likelihood, sensitivity and specificity
   • raw data from a 2x2 table
• Links to confidence interval calculator

Delfini Primer: "Evaluating Diagnostic Tests: Challenges with Measures of Test Function" online at Evidence Essentials
Download helpful one-pager that helps you better understand complexities of diagnostic testing measures »

STARD for Standards in Reporting of Diagnostic Accuracy

Click here to go to the Delfini Diagnostic Testing Tool Set.

Equivalence & Non-inferiority Trials
  1. Cleophas TJ, Zwinderman AH. Limitations of randomized clinical trials. Proposed alternative designs. Clin Chem Lab Med. 2000 Dec;38(12):1217-23. Review. PubMed PMID: 11205684.

  2. Committee for Medicinal Products for Human Use; Efficacy Working Party; Committee for Release for Consultation. Committee for Medicinal Products for Human Use (CHMP) guideline on the choice of the non-inferiority margin.  Stat Med. 2006 May 30;25(10):1628-38. PMID: 16639773

  3. D'Agostino RB Sr, Massaro JM, Sullivan LM.  Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics.  Stat Med. 2003 Jan 30;22(2):169-86. PMID: 12520555

  4. Gøtzsche PC. Lessons from and cautions about noninferiority and equivalence randomized trials. JAMA. 2006 Mar 8;295(10):1172-4. PubMed PMID: 16522840.

  5. Jones B et al.  Trials to assess equivalence:  the importance of rigorous methods.  BMJ 1996; 313:36-39 (6 July)

  6. Kirshner B.  Methodological standards for assessing therapeutic equivalence.  J Clin Epidemiol 1991. Vol. 44, No 8: 839-849

  7. Lange S, Freitag G.   Choice of delta: requirements and reality--results of a systematic review. Biom J. 2005 Feb;47(1):12-27; discussion 99-107. Review.   PMID: 16395993

  8. Le Henanff A, Giraudeau B, Baron G, Ravaud P. Quality of reporting of noninferiority and equivalence randomized trials. JAMA. 2006 Mar 8;295(10):1147-51. PubMed PMID: 16522835.

  9. Starred FavoritesPiaggio G et al. Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement JAMA. 2006;295:1152-1160. PMID: 16522836.

  10. Sanjay Kaul B et al. Good Enough: A Primer on the Analysis and Interpretation of Noninferiority Trials. Ann Intern Med. 2006;145:62-69. PMID 16522836

  11. Starred FavoritesSchumi J, Wittes JT. Through the looking glass: understanding non-inferiority trials. 2011 May 3;12:106. doi: 10.1186/1745-6215-12-106. Review. PubMed PMID: 21539749; PubMed Central PMCID: PMC3113981.
  12. Starred FavoritesSnapinn S. Noninferiority trials [commentary]. Curr Control Trials Cardiovasc Med 2000;1:19-21

  13. Wiens BL.  Choosing an equivalence limit for noninferiority or equivalence studies. Control Clin Trials. 2002 Feb;23(1):2-14. Erratum in: Control Clin Trials 2002 Dec;23(6):774. PMID: 11852160


Biostatistics for Dummies: John C. Pezzullo By John C. Pezzullo, PhD

Excepts From Our Review on Amazon

Great Help for Critical Appraisers

  • An excellent resource
  • Useful to critical appraisers because it can help us understand why certain common statistical tests are used in studies
  • Provides a needed resource for answering questions about various tests
  • Helpful explanations
  • Written in a clear style with the goal of making difficult information accessible and understandable
  • Friendly style due to author's wit and charm, and the reassurance he provides along the way

Read our full review here. Go to Amazon page and full customer reviews here.

Cost-analysis and Cost-effectiveness Analysis
BookGold MR, Siegel JE, et al., Eds. (1996)
Cost-Effectiveness in Health and Medicine. New York, Oxford University Press.

Ganiats TG
Cost-effective analysis.
Primary Care.
22(2): 307-318, 1995.

Deyo R
Cost-effectiveness of Primary Care.
J Am Board Fam Pract
13(1):47-54, 2000. PMID: 10682885

Ramsey S et al
Weighing the Economic Evidence: Guidelines for Critical Assessment of Cost-Effectiveness Analyses.
J Am Board Fam Pract
12(6): 477-485, 1999.
PubMed PMID: 10612366

Click here to see our Examples of Cost Analysis Tools.

Visual Display of Information

Starred Favorites Everything by Edward Tufte!

Click here to go to our online newsletter of examples of information and decision aids for practitioners and patients — On the Same Page ™.

General Online Articles About Medical Science

The Blog of Scott K. Abergg, MD MPH:

Why Almost Everything You Hear About Medicine is Wrong by Sharon Begley in Newsweek, January 24, 2011

The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer in The New Yorker, December 13, 2010

Related Article & Regression-to-the-Mean Explanation:
Study vs. Study: The Decline Effect and Why Scientific 'Truth' So Often Turns Out Wrong—It May Not Be So Surprising That Study Findings Don't Always Hold Up
by John Allen Paulos, ABC News/Technology, January 2, 2011

About the work of Professor John Ioannidis MD and the medical misinformation challenge:
Letter to the Editor, How to Evaluate Medical Science and "Lies, Damned Lies, and Medical Science" by David H. Freedman in The Atlantic, November 2010.


Contact UsCONTACT DELFINI Delfini Group EBM DolphinDelfini Group EBM Dolphin

Return to Top

© 2002-2020 Delfini Group, LLC. All Rights Reserved Worldwide.
Use of this website implies your agreement to our Notices.

EBM Solutions for Evidence-based Health Care Quality