A Caution When Evaluating Systematic Reviews and Meta-analyses

Status

A Caution When Evaluating Systematic Reviews and Meta-analyses

We would like to draw critical appraisers’ attention to an infrequent but important problem encountered in some systematic reviews—the accuracy of standardized mean differences in some reviews. Meta-analysis of trials that have used different scales to record outcomes of a similar nature requires data transformation to a uniform scale, the standardized mean difference (SMD). Gøtzsche and colleagues, in a review of 27 meta-analyses utilizing SMD found that a high proportion of meta-analyses based on SMDs contained meaningful errors in data extraction and calculation of point estimates.[1] Gøtzsche et al. audited two trials from each review and found that, in 17 meta-analyses (63%), there were errors for at least 1 of the 2 trials examined. We recommend that critical appraisers be aware of this issue.

1. Gøtzsche PC, Hróbjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA. 2007 Jul 25;298(4):430-7. Erratum in: JAMA. 2007 Nov 21;298(19):2264. PubMed PMID:17652297.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Tumblr Email

Improving Results Reporting in Clinical Trials: Case Study—Time-to-Event Analysis and Hazard Ratio Reporting Advice

Status

Improving Results Reporting in Clinical Trials: Case Study—Time-to-Event Analysis and Hazard Ratio Reporting Advice

We frequently see clinical trial abstracts—especially those using time-to-event analyses—that are not well-understood by readers. Fictional example for illustrative purposes:

In a 3-year randomized controlled trial (RCT) of drug A versus placebo in women with advanced breast cancer, the investigators presented their abstract results in terms of relative risk reduction for death (19%) along with the hazard ratio (hazard ratio = 0.76, 95% confidence interval [CI] 0.56 to 0.94, P = 0.04). They also stated that, “This reduction represented a 5-month improvement in median survival (24 months in the drug A group vs. 19 months in the placebo group).” Following this information, the authors stated that the three-year survival probability was 29% in the drug A group versus 21.0% in the placebo group.

Many readers do not understand hazard ratios and will conclude that a 5 month improvement in median survival is not clinically meaningful. We believe it would have been more useful to present mortality information (which the authors frequently present in  results section, but is not easily found by many readers).

A much more meaningful abstract statement would go something like this: After 3 years, the overall mortality was 59% in the drug A group compared with 68% in the placebo group which represents an absolute risk reduction (ARR) of 9%, P=0.04, number needed to treat (NNT) 11.  This information is much more impressive and much more easily understood than a 5-month increase in median survival and uses statistics familiar to clinicians.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Tumblr Email