Abstract
Objective
This article is to establish recommendations for conducting quantitative synthesis,
or meta-analysis, using study-level data in comparative effectiveness reviews (CERs)
for the Evidence-based Practice Center (EPC) program of the Agency for Healthcare
Research and Quality.
Study Design and Setting
We focused on recurrent issues in the EPC program and the recommendations were developed
using group discussion and consensus based on current knowledge in the literature.
Results
We first discussed considerations for deciding whether to combine studies, followed
by discussions on indirect comparison and incorporation of indirect evidence. Then,
we described our recommendations on choosing effect measures and statistical models,
giving special attention to combining studies with rare events; and on testing and
exploring heterogeneity. Finally, we briefly presented recommendations on combining
studies of mixed design and on sensitivity analysis.
Conclusion
Quantitative synthesis should be conducted in a transparent and consistent way. Inclusion
of multiple alternative interventions in CERs increases the complexity of quantitative
synthesis, whereas the basic issues in quantitative synthesis remain crucial considerations
in quantitative synthesis for a CER. We will cover more issues in future versions
and update and improve recommendations with the accumulation of new research to advance
the goal for transparency and consistency.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes.New Engl J Med. 2007; 356: 2457-2471
- Meta-analysis of rare events: an update and sensitivity analysis of cardiovascular events in randomized trials of rosiglitazone.Clin Trials. 2008; 5: 116-120
- Uncertain effects of rosiglitazone on the risk for myocardial infarction and cardiovascular death.Ann Intern Med. 2007; 147: 578-581
- Fixed vs random effects meta-analysis in rare event studies: the rosiglitazone link with myocardial infarction and cardiac death.Stat Med. 2007; 26: 4375-4385
- Hearing on FDA’s role in evaluating safety of avandia.(Available at) (Accessed May 31, 2010)
- Evidence-based Practice Centers.(Available at) (Accessed May 31, 2010)
- Principles for developing guidance: AHRQ and the effective health care program.J Clin Epidemiol. 2010; 63: 484-490
- Cochrane handbook for systematic reviews of interventions.(Available at) (Accessed May 31, 2010)
- Quantifying heterogeneity in a meta-analysis.Stat Med. 2002; 21: 1539-1558
- Detecting and describing heterogeneity in meta-analysis.Stat Med. 1998; 17: 841-856
- Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses.Stat Med. 2000; 19: 1707-1728
- Combination of direct and indirect evidence in mixed treatment comparisons.Stat Med. 2004; 23: 3105-3124
- Network meta-analysis for indirect treatment comparisons.Stat Med. 2002; 21: 2313-2324
- The transitive fallacy for randomized trials: if A bests B and B bests C in separate trials, is A better than C?.BMC Med Res Methodol. 2002; 2: 13
- The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials.J Clin Epidemiol. 1997; 50: 683-691
- Simultaneous comparison of multiple treatments: combining direct and indirect evidence.BMJ. 2005; 331: 897-900
- Indirect comparisons of competing interventions.Health Technol Assess. 2005; 9: 1-134
- Indirect comparison in evaluating relative efficacy illustrated by antimicrobial prophylaxis in colorectal surgery.Control Clin Trials. 2000; 21: 488-497
- Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses.Lancet. 2006; 368: 1503-1515
- Indirect comparisons: the mesh and mess of clinical trials.Lancet. 2006; 368: 1470-1472
- Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses.BMJ. 2003; 326: 472
- Meta-analysis of migraine headache treatments: combining information from heterogeneous designs.J Am Stat Assoc. 1999; 94: 16-28
- Assessing evidence inconsistency in mixed treatment comparisons.J Am Stat Assoc. 2006; 101: 447-459
- Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.Stat Med. 2002; 21: 1575-1600
- Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events.Stat Med. 2007; 26: 53-77
- The number needed to treat: a clinically useful measure of treatment effect.BMJ. 1995; 310: 452-454
- ’Unqualified success’ and ’unmitigated failure’: number-needed-to-treat-related concepts for assessing treatment efficacy in the presence of treatment-induced adverse events.Int J Epidemiol. 1996; 25: 704-712
- Distribution theory for glass’s estimator of effect size and related estimators.J Educ Stat. 1981; 6: 107-128
- Statistical power analysis for the behavioral sciences.L. Erlbaum Associates, Hillsdale, NJ1988
- Evaluation of clinically relevant changes in patient reported outcomes in knee and hip osteoarthritis: the minimal clinically important improvement.Ann Rheum Dis. 2005; 64: 29-33
- Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.Stat Med. 1998; 17: 2815-2834
- Practical methods for incorporating summary time-to-event data into meta-analysis.Trials. 2007; 8: 16
- Estimating number of events from the Kaplan-Meier curve for incorporation in a literature-based meta-analysis: what you don’t see you can’t get!.Biometrics. 2000; 56: 886-892
- Meta-analysis in clinical trials.Control Clin Trials. 1986; 7: 177-188
- Random-effects model for meta-analysis of clinical trials: an update.Contemp Clin Trials. 2007; 28: 105-114
- A comparison of statistical methods for meta-analysis.Stat Med. 2001; 20: 825-840
- Bayesian approaches to random-effects meta-analysis: a comparative study.Stat Med. 1995; 14: 2685-2699
- What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.Stat Med. 2004; 23: 1351-1375
- The exact analysis of contingency tables in medical research.Cancer Treat Res. 1995; 75: 177-202
- Exact logistic regression: theory and examples.Stat Med. 1995; 14: 2143-2160
- Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data.BMC Med Res Methodol. 2007; 7: 5
- An assessment of the use of the continuity correction for sparse data in meta-analysis.Comm Stat Simulat Comput. 1996; 25: 1031-1056
- The binomial distribution of meta-analysis was preferred to model within-study variability.J Clin Epidemiol. 2008; 61: 41-51
- How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS.Stat Med. 2005; 24: 2401-2428
- WinBUGS.(Available at) (Accessed May 31, 2010)
- Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.J Clin Epidemiol. 1995; 48 (discussion 9–60): 45-57
- Measuring inconsistency in meta-analyses.BMJ. 2003; 327: 557-560
- Uncertainty in heterogeneity estimates in meta-analyses.BMJ. 2007; 335: 914-916
- Summing up evidence: one answer is not always enough.Lancet. 1998; 351: 123-127
- An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials.Stat Med. 1998; 17: 1923-1942
- Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors.J Clin Epidemiol. 2004; 57: 683-697
- Controlling the risk of spurious findings from meta-regression.Stat Med. 2004; 23: 1663-1682
- Comparing medical interventions: AHRQ and the effective health-care program.J Clin Epidemiol. 2010; 63: 481-483
- Randomized, controlled trials, observational studies, and the hierarchy of research designs.New Engl J Med. 2000; 342: 1887-1892
- A comparison of observational studies and randomized, controlled trials.New Engl J Med. 2000; 342: 1878-1886
- Comparison of evidence of treatment effects in randomized and nonrandomized studies.JAMA. 2001; 286: 821-830
- Contradicted and initially stronger effects in highly cited clinical research.JAMA. 2005; 294: 218-228
- A new form of meta-analysis for combining results from randomized clinical trials and medical-practice databases.Int J Technol Assess Health Care. 1993; 9: 440-449
- Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening.Stat Med. 2000; 19: 3359-3376
- Re: “A critical look at some popular meta-analytic methods”.Am J Epidemiol. 1994; 140 (discussion 300–301): 297-299
Article info
Publication history
Published online: April 08, 2011
Accepted:
August 9,
2010
Footnotes
This article was written with support from the Effective Health Care Program at the U.S. Agency for Healthcare Research and Quality.
Identification
Copyright
© 2011 Elsevier Inc. Published by Elsevier Inc. All rights reserved.
