Volume — Quality of Evidence:
Secondary Sources

Newest
08/20/2014: Clinical Guidelines and Elderly Patients—Proceed with Caution

Contents      

• Reliable Clinical Guidelines—Great Idea, Not-Such-A-Great Reality
• Quality of Clinical Guidelines
• Clinical Guidelines and Elderly Patients—Proceed with Caution
• Poor Quality of Guidelines: Case Study — The Evidence on Well-Child Care Recommendations
• Another Cautionary Tale About Clinical Guidelines: Screening for Heart Attack Prevention and Education (SHAPE) Task Force Guidelines Example
• Review of Endocrinology Guidelines

Go to DelfiniClick™ for all volumes.

Reliable Clinical Guidelines—Great Idea, Not-Such-A-Great Reality
01/23/2013

Although clinical guideline recommendations about managing a given condition may differ, guidelines are, in general, considered to be important sources for individual clinical decision-making, protocol development, order sets, performance measures and insurance coverage. The Institute of Medicine [IOM] has created important recommendations that guideline developers should pay attention to—

1. Transparency;
2.  Management of conflict of interest;
3.  Guideline development group composition;
4. How the evidence review is used to inform clinical recommendations;
5.  Establishing evidence foundations for making strength of recommendation ratings;
6. Clear articulation of recommendations;
7. External review; and,
8. Updating.

Investigators recently evaluated 114 randomly chosen guidelines against a selection from the IOM standards and found poor adherence [Kung 12]. The group found that the overall median number of IOM standards satisfied was only 8 out of 18 (44.4%) of those standards. They also found that subspecialty societies tended to satisfy fewer IOM methodological standards. This study shows that there has been no change in guideline quality over the past decade and a half when an earlier study found similar results [Shaneyfeld 99].  This finding, of course, is likely to have the effect of leaving end-users uncertain as to how to best incorporate clinical guidelines into clinical practice and care improvements.  Further, Kung’s study found that few guidelines groups included information scientists (individuals skilled in critical appraisal of the evidence to determine the reliability of the results) and even fewer included patients or patient representatives.

An editorialist suggests that currently there are 5 things we need [Ransohoff]. We need:

1. An agreed-upon transparent, trustworthy process for developing ways to evaluate clinical guidelines and their recommendations.

2. A reliable method to express the degree of adherence to each IOM or other agreed-upon standard and a method for creating a composite measure of adherence.

From these two steps, we must create a “total trustworthiness score” which reflects adherence to all standards.

3. To accept that our current processes of developing trustworthy measures is a work in progress. Therefore, stakeholders must actively participate in accomplishing these 5 tasks.

4. To identify an institutional home that can sustain the process of developing measures of trustworthiness.

5. To develop a marketplace for trustworthy guidelines. Ratings should be displayed alongside each recommendation.

At this time, we have to agree with Shaneyfeld who wrote an accompanying commentary to Kung’s study [Shaneyfeld 12]:

What will the next decade of guideline development be like? I am not optimistic that much will improve. No one seems interested in curtailing the out-of-control guideline industry. Guideline developers seem set in their ways. I agree with the IOM that the Agency for Healthcare Research and Quality (AHRQ) should require guidelines to indicate their adherence to development standards. I think a necessary next step is for the AHRQ to certify guidelines that meet these standards and allow only certified guidelines to be published in the National Guidelines Clearinghouse. Currently, readers cannot rely on the fact that a guideline is published in the National Guidelines Clearinghouse as evidence of its trustworthiness, as demonstrated by Kung et al. I hope efforts by the Guidelines International Network are successful, but until then, in guidelines we cannot trust.

References

1. IOM: Graham R, Mancher M, Wolman DM,  et al; Committee on Standards for Developing Trustworthy Clinical Practice Guidelines; Board on Health Care Services.  Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011 http://www.nap.edu/catalog.php?record_id=13058

2. Kung J, Miller RR, Mackowiak PA. Failure of Clinical Practice Guidelines to Meet Institute of Medicine Standards: Two More Decades of Little, If Any, Progress. Arch Intern Med. 2012 Oct 22:1-6. doi: 10.1001/2013.jamainternmed.56. [Epub ahead of print] PubMed PMID: 23089902.

3.  Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013 Jan 9;309(2):139-40. doi: 10.1001/jama.2012.156703. PubMed PMID: 23299601.

4. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 1999 May 26;281(20):1900-5. PubMed PMID: 10349893.

5. Shaneyfelt T. In Guidelines We Cannot Trust: Comment on "Failure of Clinical Practice Guidelines to Meet Institute of Medicine Standards". Arch Intern Med. 2012 Oct 22:1-2. doi: 10.1001/2013.jamainternmed.335. [Epub ahead of print] PubMed PMID: 23089851.

Click here to share this page. If you are at an entry URL (#title), copy URL, then click "share" button to paste into body text of your message.

Clinical guidelines can improve care and conserve resources, but they vary greatly in quality. Several groups (including Delfini) have created tools to evaluate someone else’s guidelines. A key component of guideline evaluation is the process by which the guideline was developed. In a recent issue of the Journal of Clinical Epidemiology, Giannakakis et al report that although the use of RCTs in developing clinical guidelines has increased, almost half of the guidelines appearing in journals such as the Annals of Internal Medicine, BMJ, JAMA, NEJM, Lancet and Pediatrics do not cite RCTs! Because observational studies and expert opinion are usually insufficient to draw conclusions about effectiveness (cause/effect relationship between intervention and outcomes), any prevention or treatment guidelines not based on RCTs may have major problems with validity.

To read the abstract of this article, go to —

J Clin Epidemiol 2002 Jun;55(6):545-55
Citation of randomized evidence in support of guidelines of therapeutic and preventive interventions.
Giannakakis IA, Haidich AB, Contopoulos-Ioannidis DG, Papanikolaou GN, Baltogianni MS, Ioannidis JP.

http://www.ncbi.nlm.nih.gov/pubmed/12063096?dopt=Abstract

Click here to share this page. If you are at an entry URL (#title), copy URL, then click "share" button to paste into body text of your message.

Clinical Guidelines and Elderly Patients—Proceed with Caution

We were alerted to this important study of clinical practice guidelines (CPGs) in elderly patients with co-morbidities [1] by Demetra Antimisiaris, an associate professor who directs the polypharmacy initiative at the University of Louisville School of Medicine.
The authors report that most CPGs did not modify recommendations for older patients with multiple comorbidities. Most also did not comment the quality of the underlying scientific evidence or provide guidance for incorporating patient preferences into treatment plans. If the relevant CPGs were followed, one hypothetical patient described in the report would be prescribed 12 medications (costing her $406 per month). Use of guidelines for this patient would result in a complicated medication schedule, and adverse events would be likely. The authors state that, "Although CPGs provide detailed guidance for managing single diseases, they fail to address the needs of older patients with complex comorbid illness." CPGs rarely address treatment of patients with 3 or more chronic diseases—a group that includes half of the population older than 65 years. Adhering to current CPGs in caring for an older person with several comorbidities may have multiple undesirable effects and could result in net harms.

1. Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA. 2005 Aug 10;294(6):716-24. PubMed PMID: 16091574.

Click here to share this page. If you are at an entry URL (#title), copy URL, then click "share" button to paste into body text of your message.

Poor Quality of Guidelines: Case Study — The Evidence on Well-Child Care Recommendations

An article in the December 2004 issue of PEDIATRICS illustrates a common problem in health care—the gap between expert’s opinions and recommendations and quality scientific medical evidence. Moyer and Butler reviewed recommendations for well-child care made by seven major North American organizations and compared them to the best available evidence (systematic reviews of randomized controlled trials and individual randomized controlled trials).

Results
Forty-two preventive interventions were recommended by two or more of these organizations; however, clinical trials have been conducted for only two of these recommended screening interventions.

Some of the key points made by the authors are --

• There is evidence to support the use of folate to prevent neural tube defects.
• No trials were found supporting the other recommended prophylactic interventions.
• There is some evidence that intensive counseling can change some health risk behaviors -- repeated intensive counseling is most likely to be effective. However, some behavioral counseling interventions were shown to have harmful effects.
• Trial data regarding benefits of iron supplementation for development outcomes is lacking.

Comments
This article is useful in that it points out the ongoing problem of making recommendations which are not based on valid, relevant evidence. The authors also note that implementing unproven interventions can result in harm and wasted resources. They make a plea for including the supporting evidence when making recommendations.

Moyer VA and Butler M. Gaps in the Evidence for Well-Child Care: A Challenge to Our Profession. PEDIATRICS;114 No. 6 : 1511-1521. PMID: 15574609

Click here to share this page. If you are at an entry URL (#title), copy URL, then click "share" button to paste into body text of your message.

Another Cautionary Tale About Clinical Guidelines: Screening for Heart Attack Prevention and Education (SHAPE) Task Force Guidelines Example
03/23/08

The Screening for Heart Attack Prevention and Education (SHAPE) Task Force guidelines[1] are clinical recommendations developed through a consensus process by a group of health care professionals and others. The lead author of the SHAPE guidelines is the CEO and President of Endothelix. Endothilix is medical device company “dedicated to human vascular health, with particular emphasis on screening for and monitoring vascular endothelial dysfunction.”[2] There are several unique features of these guidelines. They go far beyond any other cardiovascular risk screening guidelines and “call for” the following:

• Noninvasive screening of all asymptomatic men 45–75 years of age and asymptomatic women 55–75 years of age (except those defined as very low risk) to detect and treat those with subclinical atherosclerosis through…
• A variety of screening tests including measurement of coronary artery calcification by computed tomography scanning and carotid artery intima–media thickness and plaque by ultrasonography.

The authors recommend careful and responsible implementation of these tests as part of a comprehensive risk assessment and reduction approach. They do not, however, provide any evidence of improved outcomes for patients undergoing these tests and state that cost-effectiveness of the use of these tests in a comprehensive strategy must be validated.

In his commentary on the SHAPE Task Force guidelines in the January 9, 2008 issue of the Journal of the American Medical Association (JAMA)[3], Peter Jacobson points out that SHAPE’s recommendations are bad public policy, in part, because they lack evidence for improved outcomes and the possibility of harms outweighing benefits. Jacobson suggests that guidelines be “vetted” by professional societies as a way of establishing validity and clinical usefulness.

At first blush this sounds reasonable. But unfortunately this will not solve the problem of unproven clinical recommendations. We have seen numerous examples of guidelines published by professional societies which are based on weak or fatally flawed evidence such as in the areas of cholesterol screening in children, treatment of Bell’s Palsy and interventions for chronic spinal pain. Others have reported similar findings.[4] Grilli et al.[5], in a review of 431 guidelines produced by U.S. medical societies, found that —

• 87% did not report whether a systematic search of the literature was performed;
• 82% did not apply explicit criteria to grade their evidence;
• 67% did not describe the type of professionals involved in the development of the guideline.
• Acceptance of unproven interventions is partly due to a fairly universal lack of effective training for the health professions in understanding scientific validity.

We recommend that users of guidelines and other clinical recommendations scrutinize all such documents for validity and usefulness rather than relying on endorsement by professional societies, reliance upon which is reasonably likely to lead to outcomes as problematic as Jacobsen suggests.

At a minimum, we suggest the pertinent questions are these:

1. Are the recommendations rigorously evidence-based and their development transparent? This requires understanding principles of scientific validity and should include performing a critical appraisal audit of the science upon which the recommendations are based. There are many resources available to help readers evaluate information sources for validity.
2. Is this information relevant to patients’ needs? Are the expected outcomes clinically significant and will they provide reasonable estimates of benefit? Are the important recommendations/options (with benefits, risks, harms, uncertainties, alternatives and costs of each choice) provided? Do the recommendations accommodate differing patient values and preferences?
3. Can this improvement be implemented and is it likely to succeed? How will the guideline impact outcomes in the setting in which it is applied? Can one measure the effect of implementation?
4. How current is this document?
5. Who developed the recommendations? Were epidemiologic and clinical perspectives included? Were other disciplines and perspectives represented as needed?
6. Are any limitations described?
7. Are there ethical issues to be considered?[6]

Although peer-review may be desirable, we believe that the type of checklist above provides a more appropriate solution for the evaluation of clinical guidelines for validity and clinical usefulness than specialty society endorsement because it helps remove the bias which may be present in any group making clinical recommendations based on consensus and low quality evidence.[7] We also wholeheartedly agree that “guidelines” should not be mandated through legislation. In the case of the SHAPE recommendations, Texas House Representative Rene Oliveira plans to introduce in 2008 his bill into state legislature. The bill would mandate insurers to cover screening of asymptomatic atherosclerosis using calcium scanning and carotid ultrasound, as recommended in the SHAPE initiative. Let’s hope the Texas legislature is awake and alert.

1. Naghavi M, Falk E, Hecht HS, et al. From vulnerable plaque to vulnerable patient, part III: executive summary of the Screening for Heart Attack Prevention and Education (SHAPE) Task Force Report. Am J Cardiol. 2006;98(2A):2H-15H.
2. http://www.shapesociety.org/ Accessed 04/09/2013.
3. Jacobson PD. Transforming clinical practice guidelines into legislative mandates. Proceed with abundant caution. JAMA 2008; 299:208-210.
4. Giannakakis IA, Haidich AB, Contopoulos-Ioannidis DG, Papanikolaou GN, Baltogianni MS, Ioannidis JP. Citation of randomized evidence in support of guidelines of therapeutic and preventive interventions. J Clin Epidemiol. 2002 Jun;55(6):545-55.
5. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000 Jan 8;355(9198):103-6.
6. Adapted from Project Appraisal Tool. http://www.delfini.org/delfiniTools.htm. Accessed 1/15/08.
7. Kaptchuk TJ. Effect of interpretive bias on research evidence. BMJ. 2003 Sep 27;327(7417):752.

Click here to share this page. If you are at an entry URL (#title), copy URL, then click "share" button to paste into body text of your message.

Review of Endocrinology Guidelines
05/14/2013

Decision-makers frequently rely on the body of pertinent research in making decisions regarding clinical management decisions. The goal is to critically appraise and synthesize the evidence before making recommendations, developing protocols and making other decisions. Serious attention is paid to the validity of the primary studies to determine reliability before accepting them into the review.  Brito and colleagues have described the rigor of systematic reviews (SRs) cited from 2006 until January 2012 in support of the clinical practice guidelines put forth by the Endocrine Society using the Assessment of Multiple Systematic Reviews (AMSTAR) tool [1].

The authors included 69 of 2817 studies. These 69 SRs had a mean AMSTAR score of 6.4 (standard deviation, 2.5) of a maximum score of 11, with scores improving over time. Thirty five percent of the included SRs were of low-quality (methodological AMSTAR score 1 or 2 of 5, and were cited in 24 different recommendations). These low quality SRs were the main evidentiary support for five recommendations, of which only one acknowledged the quality of SRs.

The authors conclude that few recommendations in field of endocrinology are supported by reliable SRs and that the quality of the endocrinology SRs is suboptimal and is currently not being addressed by guideline developers. SRs should reliably represent the body of relevant evidence.  The authors urge authors and journal editors to pay attention to bias and adequate reporting.

Delfini note: Once again we see a review of guideline work which suggests using caution in accepting clinical recommendations without critical appraisal of the evidence and knowing the strength of the evidence supporting clinical recommendations.

1. Brito JP, Tsapas A, Griebeler ML, Wang Z, Prutsky GJ, Domecq JP, Murad MH, Montori VM. Systematic reviews supporting practice guideline recommendations lack protection against bias. J Clin Epidemiol. 2013 Jun;66(6):633-8. doi: 10.1016/j.jclinepi.2013.01.008. Epub 2013 Mar 16. PubMed PMID: 23510557.

Click here to share this page. If you are at an entry URL (#title), copy URL, then click "share" button to paste into body text of your message.

DelfiniClick™