Cochrane HIV/AIDS Group's Editorial Policy

Dealing with Heterogeneity and Statistical Analysis

  • Reviewers should routinely test for statistical heterogeneity (Petitti 2001; Thompson 2001). Tests for
    heterogeneity, in general, have low power (Hedges 2001). Therefore, the p-value may need to be
    conservative (p <0.1 or p <0.2) for the chi-square or Q test. Reviewers should report the actual results of the test statistic, not as significant/not significant, and state the p-value cut point used for rejecting the null hypothesis (Petitti 2001).
  • Heterogeneity is a study finding. Since we can't know in advance whether we are going to find
    heterogeneity, reviewers could do the analysis using both random and fixed effects models and then use this information as a sensitivity analysis (Villar 2001; Petitti 2001). When the analyses are done using both random and fixed effects models and conclusions differ depending on the model, reviewers must state the rationale for basing their conclusions on one or the other model (Petitti 2001).
  • If significant heterogeneity is found, reviewers may formally explore possible reasons for
    heterogeneity, using detailed tabulation, graphical displays, or techniques such as meta-regression
    (Thompson 2001; Deeks 2001). It is preferable that reviewers pre-specify a priori in the protocol what
    factors they might want to consider as explanations of heterogeneity (factors they may want to use for
    stratified or sensitivity analyses) (Villar 2001). Pre-specifying this will reduce problems in interpreting
    analyses based on post-hoc stratification.
  • In some situations (i.e., very significant heterogeneity) heterogeneity may deserve more emphasis
    than the summary estimates (Thompson 2001; CRH 2001; Glasziou 2001). In fact, reviewers may
    choose not to force the results into a single summary estimate. If a summary estimate is reported,
    reviewers need to warn the readers about interpreting the summary estimates in the presence of
    heterogeneity. In fact, in some situations, the conclusions of the review may change considerably if
    heterogeneity is successfully explained (Thompson 2001).

References:

  1. Thompson SG. Why and how sources of heterogeneity should be investigated. In: Egger et al
    (editors). Systematic reviews in health care. BMJ Books, 2001. p 157-175.
  2. Deeks et al. Statistical methods for examining heterogeneity and combing results from several studies
    in meta-analysis. Systematic reviews in health care. BMJ Books, 2001. p 285-312.
  3. Cochrane Reviewers' Handbook. 4.1.4. October, 2001.
  4. Glasziou P et al. Systematic reviews in health care. A practical guide. Cambridge University Press,
    2001
  5. Hedges LV, Pigott TD. The power of statistical tests in meta-analysis. Psychol Methods 2001;6
    (3):203-17.
  6. Petitti DB. Approaches to heterogeneity in meta-analyses. Statistics in Medicine 2001;20:3625-33.
  7. Villar J, Mackey ME, Carroli G, Donner A. Meta-analyses in systematic reviews of randomized
    controlled trials in perinatal medicine: comparison of fixed and random effects models. Statistics in Medicine 2001;20:3635-47.

 

Copyright © 2007. The Regents of the University of California. All rights reserved.