• HOME
• ABOUT US
• NEWS & INFO
• OUR REVIEWS
• ROADMAP
• MEETINGS
• PUBLICATIONS
• GET INVOLVED
• FORMS
• LINKS

Editorial Policy

Inclusion and Appraisal of Experimental and Non-experimental (Observational) Studies

Although randomized controlled trials (RCTs) are considered the most rigorous study design to assess the effectiveness of interventions, many important topics in HIV/AIDS prevention and treatment do not lend themselves to being evaluated through RCTs or have not yet been evaluated with RCTs. As it is the goal of the Cochrane HIV/AIDS Group to assess all evidence of HIV/AIDS prevention and treatment, we must sometimes gather and assess evidence of both experimental and observational studies. For reviews that will include both experimental and non-experimental studies, it is important to have a clear set of a priori definitions regarding categorization of different types of study designs. Additionally, authors should consider in the protocol how to combine and interpret the results of experimental and observational studies. We encourage reviewers to create separate categories by study design and combine data only as appropriate (see # 5 below).

Assessment of quality of observational studies is more difficult than assessment of quality of RCTs and other experimental studies. Quality assessment methods for observational studies have not been adequately developed, and, although several assessment scales and checklists are used, none of them have been fully validated or shown to include criteria that are associated with the effect size (outcome) in empirical studies. Some general guidelines and suggestions to provide assistance are provided here:

1. Appraising observational studies with checklists/scales designed for RCTs may not be appropriate.
2. There are several checklists/scales targeted towards observational studies available. Please consider using one of the following that is the most appropriate for the studies you are including:
   • The "Newcastle-Ottawa Scale (NOS) for assessing the quality of non-randomized studies in meta-analyses" appears quite comprehensive and this instrument has been partly validated (Wells et al). The NOS instrument is available from this URL: http://www.lri.ca/programs/ceu/oxford.htm. The web site has two checklists (one for cohort studies and another for case-control studies) and also a manual for coding. (This is the scale recommended by the Cochrane Non-Randomized Studies Methods Working Group.)
   • If prognosis studies are being evaluated, the criteria proposed by Altman (2001) may be useful. This list of comprehensive criteria, however, is not validated.
   • The Agency for Healthcare Research and Quality (AHRQ) in the US has recently published a document titled "Systems to Rate the Strength of Scientific Evidence," which is available at http://www.ahrq.gov/clinic/epcix.htm. This document contains checklists for appraising the quality of RCTs, observational studies, and systematic reviews. According to this report, the key domains for quality assessment in observational studies are: comparability of subjects, measurement of exposure or intervention, measurement of outcomes, statistical analysis, and funding or sponsorship.
3. We suggest the use of one of these (or similar) instruments. A complete description of the instrument should be provided in the review. At least two independent reviewers should perform the quality assessment. Mechanisms for resolving discrepancies in coding should be mentioned in the protocol and the full review.
4. In general, empirical research has shown that quality scores (numeric scores based on arbitrary weights given to each item in a scale) are arbitrary, unreliable, and hard to interpret (Juni 1999, Greenland 1994). Our suggestion, therefore, is not to use quality scores. We suggest the approach of using individual components of a checklist, and ratings such as “met, partially met, not met.”
5. Once quality is assessed using individual components of one of the available checklists, the influence of quality on effect estimates (summary odds ratio, etc.) could be evaluated by subgroup analysis, stratifying by criteria met or not met. Separate summary effect estimates can be generated for studies that meet and do not meet the individual quality criterion. Only when a large number of studies are identified for inclusion, approaches such as meta-regression might be useful. The meta-regression analysis models the outcome (odds ratio, for example) of each study as the dependent variable and will include quality variables as co-variates (independent variables). Incorporating quality scores in the analysis as weights is not recommended.
6. Lastly, in reviews that include both experimental and observational studies, summary results should be presented separately for each of these two broad categories. After stratifying the results by RCTs and non-RCTs, it is recommended that stratification be done further within observational designs by study design (i.e. cohort, case-control, cross-sectional, etc), exploring potential heterogeneity due to study designs. In the discussion section, the reviewers should discuss the consistency, or lack thereof, in the results of experimental and observational studies.

NOTE: These suggestions are tentative and will be revised when the Cochrane Non-Randomized Studies Methods Group makes available its guidelines for inclusion of non-randomized studies in Cochrane reviews.

References:

• Altman DG. Systematic reviews of prognostic variables. BMJ 2001; 323: 224-228.
• Wells GA , Shea B, O'Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomized studies in meta-analyses. Department of Epidemiology and Community Medicine, University of Ottawa, Canada. http://www.lri.ca/programs/ceu/oxford.htm
• Agency for Healthcare Research and Quality. Systems to Rate the Strength of Scientific Evidence. AHRQ Evidence-based Practice Report/Technology Assessment No. 47; AHRQ Publication No. 02-E016. US Department of Health and Human Services, USA, 2002. http://www.ahrq.gov/clinic/epcix.htm
• Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for metaanalysis. JAMA 1999; 282(11): 1054-60.
• Greenland S. Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol 1994; 140(3): 290-6.