- •Quasi-experimental designs (QEDs), also called nonrandomized studies of intervention effects, can provide evidence that is both internally and externally valid for decision making.
- •There are stronger theoretical and empirical reasons to incorporate QEDs in systematic reviews of intervention effects than are commonly acknowledged.
What this adds to what is known
- •Systematic reviews of intervention effects should usually incorporate appropriately critically-appraised evidence from QEDs.
What is the implication, what should change now
- Saldanha I.J.
- Adam G.P.
- Bañez L.L.
- Bass E.B.
- Berliner E.
- Devine B.
- et al.
- 1.The external validity argument: QEDs provide evidence on the effectiveness of interventions in a wide variety of fields (e.g., health, social policy, economic development, and the environment). One reason why they are invaluable is that they tend to provide evidence of intervention effects conducted under circumstances of usual treatment practice, rather than in circumstances where allocation has been modified by researchers or treatments implemented with a higher degree of fidelity than is usually seen in practice. That is, QEDs provide evidence on treatment effectiveness under “real world” conditions, rather than treatment efficacy as a proof-of-concept. Evidence on intervention effectiveness is important for decision makers, who, in most instances, are interested not just in the population average effect, but in understanding the variability in findings (for whom, in what circumstances, why). A key advantage of incorporating QEDs in meta-analysis is that they can facilitate exploration of this variability. For example, network meta-analysis will frequently include open loops when restricted to RCTs; adding in appropriately critically appraised QEDs may provide the missing connectivity evidence to close these loops. As is the case with RCTs, the critical appraisal of QEDs is especially important to avoid incorporating biased estimates, which sometimes operate to produce overly optimistic effects.
- 2.The internal validity argument (theoretical): Saldanha et al. note threats to the internal validity of nonrandomized studies, including confounding, selection and misclassification biases, but there are also threats to validity in prospective studies like RCTs, which are not applicable to observational studies that are designed retrospectively. An example is motivational effects: when study participants are aware they are being observed as part of trial, this might affect their motivation, either to improve their performance or to demoralize them []. Thus, inclusion of different study designs could allow a triangulation approach to assessment of internal validity, where different designs contribute different information to different elements of both internal and external validity. Bradford-Hill [] referred to this as “consistency.”
- 3.The internal validity argument (empirical): the evidence cited by Saldanha et al. to support inclusion of nonrandomized studies is from meta-epidemiology. This evidence is susceptible to confounding–by population, intervention, comparator, outcome and setting–because the comparisons are indirect. In other words, the comparisons made of RCTs and NRSI/QEDs in these meta-analyses are from external replications that do not necessarily use the same populations at the same time periods or the same intervention and counterfactual conditions. Evidence on empirical validation from internal replication studies (also called within-study comparisons) is thought more credible as it directly compares the QED and RCT findings in the same populations, time-periods and settings. Systematic reviews and meta-analyses of these studies (e.g., [,]) support the internal validity of NRSI/QEDs, especially for strong designs like RDD. These studies can also provide empirical bias correction factors for different approaches []. However, there is a need for more internal replication studies from different fields, and more syntheses of these studies given the literature that exists already–see Shadish et al. [
Absolute and relative bias in eight common observational study designs: evidence from a meta-analysis. Working paper.] or, for an example in health systems, Fretheim et al. [].
- Zurovac J.
- Cook T.D.
- Deke J.
- Finucane M.M.
- Chaplin D.D.
- Coopersmith J.S.
- et al.
Campbell systematic reviews: policies and guidelines. Version 1.8.
- Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the agency for Healthcare research and quality effective Healthcare program.J Clin Epidemiol. 2022; https://doi.org/10.1016/j.jclinepi.2022.08.015
- ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.Br Med J. 2016; 355: i4919
- Experimental and quasi-experimental designs for generalized causal inference.Wadsworth CENGAGE Learning, Belmont, CA2002
- The future of education research at IES: Advancing an equity-oriented science.The National Academies Press, Washington, DC2022
- Quasi-experimental study designs series-paper 4: uses and value.J Clin Epidemiol. 2017; 89: 21-29
- The environment and disease: association or causation?.Proc R Soc Med. 1965; 58: 295-300
- The internal and external validity of the regression discontinuity design: a meta-analysis of 15 within-study comparisons: methods for policy analysis.J Policy Anal Manage. 2018; 37: 403-429
- Can non-randomised studies of interventions provide unbiased effect estimates? A systematic review of internal replication studies.Eval Rev. 2022; (193841X221116721)
- Absolute and relative bias in eight common observational study designs: evidence from a meta-analysis. Working paper.(Available at)
- Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments.J Am Stat Assoc. 2008; 103: 1334-1344
- Interrupted time-series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result.J Clin Epidemiol. 2013; 66: 883-887
- PROTOCOL: water, sanitation and hygiene (WASH) for reducing childhood mortality in low- and middle-income countries.Campbell Syst Rev. 2020; 17: e1135
- Mass deworming to improve developmental health and wellbeing of children in low-income and middle-income countries: a systematic review and network meta-analysis.Lancet Glob Health. 2017; 5: e40-e50
- Campbell systematic reviews: policies and guidelines. Version 1.8.(Available at)https://onlinelibrary.wiley.com/pb-assets/assets/18911803/Campbell%20Policies%20and%20Guidelines%20_May3%202022-1653054593497.pdfDate: 2021Date accessed: September 27, 2022
Publication stageIn Press Journal Pre-Proof
Declaration: The opinions are our own and do not necessarily represent the views of the Campbell Collaboration.
Conflict of interest: The authors are not aware of any interests, financial or otherwise, which might affect our commentary.
User LicenseCreative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) |
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy