Study Design and Setting
- •We developed guidance consisting of nine considerations to inform decisions regarding the inclusion of nonrandomized studies of interventions (NRSIs) in systematic reviews (SRs) of interventions. The considerations were related to topic scoping and refinement (n = 4), SR team formation (n = 1), and protocol development (n = 4).
- •This guidance updates the 2010 guidance provided by the Agency for Health Care Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program that recommended NRSI inclusion only if RCTs are insufficient to address the key question.
What this adds to what was known?
- •This guidance may improve the applicability of an SR's findings and their utility for end-users, but is likely to require more time, methodological expertise, and resources than would have been needed if NRSIs were excluded from the SR. Systematic reviewers should be transparent in justifying decisions regarding the inclusion of NRSIs in the SR protocol and the final publication.
What is the implication and what should change now?
1.1 Strengths and limitations of randomized controlled trials
1.2 Defining nonrandomized studies of interventions and their types
- 1)presence of a comparison group receiving a different intervention or not receiving an intervention (controlled vs. uncontrolled/single group);
- 2)experimental nature (experimental [i.e., investigator assigns group] vs. nonexperimental);
- 3)type of control group (historic control vs. concurrent control vs. none);
- 4)presence of follow-up over time (yes [i.e., longitudinal] vs. no [i.e., cross-sectional]); and
- 5)temporality, in the case of longitudinal studies (prospective vs. retrospective).
|Systematic review step||Considerations|
|Topic scoping and refinement|
Consider the following questions at a minimum when evaluating the potential utility of RCTs and NRSIs:
|Systematic review team formation|
When NRSIs are included:
The protocol should:
1.3 Threats to the internal validity of nonrandomized studies of interventions
1.3.1 Selection bias
1.4 Design and analytical approaches to address threats to validity of nonrandomized studies of interventions
3.1.1 Considerations during topic scoping and refinement
- Consideration 1: What are the decisional dilemmas and key questions being addressed, and how will the end user(s) of the SR use the evidence to inform decision-making?
- Consideration 2: Is it logical and likely for RCTs to have addressed the key questions adequately?
188.8.131.52 Interventions and Comparators
- Consideration 3: How serious is the risk of bias in NRSIs that address the key questions likely to be?
- Consideration 4: To what extent are NRSIs and RCTs likely to complement each other?
3.1.2 Consideration during systematic review team formation
- Consideration 5: As noted in Table 1, when NRSIs are planned to be included, the SR team should include members who are familiar with topic-specific data source considerations and advanced analytic methods for NRSIs.
3.1.3 Considerations during protocol development
- Consideration 6: Systematic reviewers should specify in the protocol the study design methods or features that will be eligible for the SR.
- Consideration 7: The decision to include or exclude NRSIs should be noted and explained.
- Consideration 8: Systematic reviewers should also discuss the potential implications of the decision to include or exclude NRSIs.
- Consideration 9: If NRSIs are going to be included, the protocol should also describe the processes for synthesizing evidence from RCTs and NRSIs and determining the overall strength of evidence.
4.1 Summary of guidance
4.2 Challenges during guidance development
4.3 Implications and limitations
- Evaluating non-randomised intervention studies.Health Technol Assess. 2003; 7 (iii-x): 1-173
- A structural approach to selection bias.Epidemiology. 2004; 15: 615-625
- ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.BMJ. 2016; 355: i4919
- Inclusion of non-randomized studies of interventions in systematic reviews of intervention effectiveness: an update.Agency for Healthcare Research and Quality (US), Rockville (MD)2022
- Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions-a taxonomy without labels.J Clin Epidemiol. 2017; 89: 30-42
- Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy.J Clin Epidemiol. 2011; 64: 861-871
- Including non-randomized studies on intervention effects.(Version 6.2 (updated February 2021) ed.)in: Higgins J.P.T. Thomas J. Chandler J. Cochrane handbook for systematic reviews of interventions. Cochrane, Chichester, UK2021 ([Chapter 24])
- Gordis epidemiology.6th ed. Elsevier, Philadelphia, PA2018
- Nonrandomized real-world evidence to support regulatory decision making: process for a randomized trial replication project.Clin Pharmacol Ther. 2020; 107: 817-826
- When can nonrandomized studies support valid inference regarding effectiveness or safety of new medical treatments?.Clin Pharmacol Ther. 2022; 111: 108-115
- Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.Cochrane Database Syst Rev. 2014; 2014: Mr000034
- Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study.BMC Med. 2022; 20: 174
- Identification of threshold for large (dramatic) effects that would obviate randomized trials is not possible.J Clin Epidemiol. 2022; 145: 101-111
- Agreement of treatment effects from observational studies and randomized controlled trials evaluating hydroxychloroquine, lopinavir-ritonavir, or dexamethasone for covid-19: meta-epidemiological study.BMJ. 2022; 377: e069400
- Antidepressant response in major depressive disorder: a meta-regression comparison of randomized controlled trials and observational studies.PLoS One. 2011; 6: e20811
- Concordance between the results of randomized and non-randomized interventional clinical trials assessing the efficacy of drugs for COVID-19: a cross-sectional study.J Antimicrob Chemother. 2021; 76: 2415-2418
- Evaluating the use of nonrandomized real-world data analyses for regulatory decision making.Clin Pharmacol Ther. 2019; 105: 867-877
- Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the RCT DUPLICATE initiative.Circulation. 2021; 143: 1002-1013
- Selecting observational studies for comparing medical interventions. Methods guide for effectiveness and comparative effectiveness reviews.AHRQ Methods for Effective Health Care, Rockville (MD)2010
- Observational studies in systematic reviews of comparative effectiveness: AHRQ and the Effective Health Care Program [corrected].J Clin Epidemiol. 2011; 64: 1178-1186
- A framework for “best evidence” approaches in systematic reviews.Agency for Healthcare Research and Quality, Rockville, MD2011
- Interventional treatments for acute and chronic pain: systematic review. Comparative effectiveness review no. 247.Agency for Healthcare Research and Quality, Rockville, MD2021
Publication stageIn Press Journal Pre-Proof
Funding Statement: This work was funded by the Agency for Health Care Research and Quality through the following contracts: Scientific Resource Center ( 290-2017-00003-C ), Brown University EPC ( 75Q80120D00001 ), ECRI Institute-Penn Medicine EPC ( 75Q80120D00002 ), Johns Hopkins University EPC ( 75Q80120D00003 ), Mayo Clinic EPC ( 75Q80120D00005 ), Minnesota EPC ( 75Q80120D00008 ), Pacific Northwest EPC ( 75Q80120D00006 ), RTI-University of North Carolina EPC ( 75Q80120D00007 ), and Southern California EPC ( 75Q80120D00009 ). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or the U.S. Department of Health and Human Services.
Disclosure: The authors have no relevant financial relationships to disclose.
Conflicts of interest: The authors have no conflicts of interest to disclose.
User LicenseCreative Commons Attribution (CC BY 4.0) |
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
Elsevier's open access license policy