Advertisement
AHRQ Series Part II: Methods Guide for Comparative Effectiveness - Guest Editor, Mark Helfand| Volume 64, ISSUE 11, P1187-1197, November 2011

Download started.

Ok

Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program

      Abstract

      Objective

      This article is to establish recommendations for conducting quantitative synthesis, or meta-analysis, using study-level data in comparative effectiveness reviews (CERs) for the Evidence-based Practice Center (EPC) program of the Agency for Healthcare Research and Quality.

      Study Design and Setting

      We focused on recurrent issues in the EPC program and the recommendations were developed using group discussion and consensus based on current knowledge in the literature.

      Results

      We first discussed considerations for deciding whether to combine studies, followed by discussions on indirect comparison and incorporation of indirect evidence. Then, we described our recommendations on choosing effect measures and statistical models, giving special attention to combining studies with rare events; and on testing and exploring heterogeneity. Finally, we briefly presented recommendations on combining studies of mixed design and on sensitivity analysis.

      Conclusion

      Quantitative synthesis should be conducted in a transparent and consistent way. Inclusion of multiple alternative interventions in CERs increases the complexity of quantitative synthesis, whereas the basic issues in quantitative synthesis remain crucial considerations in quantitative synthesis for a CER. We will cover more issues in future versions and update and improve recommendations with the accumulation of new research to advance the goal for transparency and consistency.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Nissen S.E.
        • Wolski K.
        • Nissen S.E.
        • Wolski K.
        Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes.
        New Engl J Med. 2007; 356: 2457-2471
        • Dahabreh I.J.
        • Economopoulos K.
        • Dahabreh I.J.
        Meta-analysis of rare events: an update and sensitivity analysis of cardiovascular events in randomized trials of rosiglitazone.
        Clin Trials. 2008; 5: 116-120
        • Diamond G.A.
        • Bax L.
        • Kaul S.
        • Diamond G.A.
        • Bax L.
        • Kaul S.
        Uncertain effects of rosiglitazone on the risk for myocardial infarction and cardiovascular death.
        Ann Intern Med. 2007; 147: 578-581
        • Shuster J.J.
        • Jones L.S.
        • Salmon D.A.
        • Shuster J.J.
        • Jones L.S.
        • Salmon D.A.
        Fixed vs random effects meta-analysis in rare event studies: the rosiglitazone link with myocardial infarction and cardiac death.
        Stat Med. 2007; 26: 4375-4385
        • Committee on Oversight and Government Reform
        Hearing on FDA’s role in evaluating safety of avandia.
        (Available at) (Accessed May 31, 2010)
        • Agency for Healthcare Research and Quality (AHRQ)
        Evidence-based Practice Centers.
        (Available at) (Accessed May 31, 2010)
        • Helfand M.
        • Balshem H.
        Principles for developing guidance: AHRQ and the effective health care program.
        J Clin Epidemiol. 2010; 63: 484-490
        • Higgins J.
        Cochrane handbook for systematic reviews of interventions.
        (Available at) (Accessed May 31, 2010)
        • Higgins J.P.
        • Thompson S.G.
        Quantifying heterogeneity in a meta-analysis.
        Stat Med. 2002; 21: 1539-1558
        • Hardy R.J.
        • Thompson S.G.
        Detecting and describing heterogeneity in meta-analysis.
        Stat Med. 1998; 17: 841-856
        • Engels E.A.
        • Schmid C.H.
        • Terrin N.
        • Olkin I.
        • Lau J.
        Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses.
        Stat Med. 2000; 19: 1707-1728
        • Lu G.
        • Ades A.E.
        Combination of direct and indirect evidence in mixed treatment comparisons.
        Stat Med. 2004; 23: 3105-3124
        • Lumley T.
        Network meta-analysis for indirect treatment comparisons.
        Stat Med. 2002; 21: 2313-2324
        • Baker S.G.
        • Kramer B.S.
        The transitive fallacy for randomized trials: if A bests B and B bests C in separate trials, is A better than C?.
        BMC Med Res Methodol. 2002; 2: 13
        • Bucher H.C.
        • Guyatt G.H.
        • Griffith L.E.
        • Walter S.D.
        The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials.
        J Clin Epidemiol. 1997; 50: 683-691
        • Caldwell D.M.
        • Ades A.E.
        • Higgins J.P.
        Simultaneous comparison of multiple treatments: combining direct and indirect evidence.
        BMJ. 2005; 331: 897-900
        • Glenny A.M.
        • Altman D.G.
        • Song F.
        • Sakarovitch C.
        • Deeks J.J.
        • D’Amico R.
        • et al.
        Indirect comparisons of competing interventions.
        Health Technol Assess. 2005; 9: 1-134
        • Song F.
        • Glenny A.M.
        • Altman D.G.
        Indirect comparison in evaluating relative efficacy illustrated by antimicrobial prophylaxis in colorectal surgery.
        Control Clin Trials. 2000; 21: 488-497
        • Chou R.
        • Fu R.
        • Huffman L.H.
        • Korthuis P.T.
        • Chou R.
        • Fu R.
        • et al.
        Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses.
        Lancet. 2006; 368: 1503-1515
        • Ioannidis J.P.
        Indirect comparisons: the mesh and mess of clinical trials.
        Lancet. 2006; 368: 1470-1472
        • Song F.
        • Altman D.G.
        • Glenny A.M.
        • Deeks J.J.
        • Song F.
        • Altman D.G.
        • et al.
        Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses.
        BMJ. 2003; 326: 472
        • Dominici F.
        • Parmigiani G.
        • Wolpert R.
        • Hasselblad V.
        Meta-analysis of migraine headache treatments: combining information from heterogeneous designs.
        J Am Stat Assoc. 1999; 94: 16-28
        • Lu G.
        • Ades A.
        Assessing evidence inconsistency in mixed treatment comparisons.
        J Am Stat Assoc. 2006; 101: 447-459
        • Deeks J.J.
        Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.
        Stat Med. 2002; 21: 1575-1600
        • Bradburn M.J.
        • Deeks J.J.
        • Berlin J.A.
        • Russell Localio A.
        Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events.
        Stat Med. 2007; 26: 53-77
        • Cook R.J.
        • Sackett D.L.
        The number needed to treat: a clinically useful measure of treatment effect.
        BMJ. 1995; 310: 452-454
        • Schulzer M.
        • Mancini G.B.
        ’Unqualified success’ and ’unmitigated failure’: number-needed-to-treat-related concepts for assessing treatment efficacy in the presence of treatment-induced adverse events.
        Int J Epidemiol. 1996; 25: 704-712
        • Hedges L.V.
        Distribution theory for glass’s estimator of effect size and related estimators.
        J Educ Stat. 1981; 6: 107-128
        • Cohen J.
        Statistical power analysis for the behavioral sciences.
        L. Erlbaum Associates, Hillsdale, NJ1988
        • Tubach F.
        • Ravaud P.
        • Baron G.
        • Falissard B.
        • Logeart I.
        • Bellamy N.
        • et al.
        Evaluation of clinically relevant changes in patient reported outcomes in knee and hip osteoarthritis: the minimal clinically important improvement.
        Ann Rheum Dis. 2005; 64: 29-33
        • Parmar M.K.
        • Torri V.
        • Stewart L.
        Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.
        Stat Med. 1998; 17: 2815-2834
        • Tierney J.
        • Stewart L.
        • Ghersi D.
        • Burdett S.
        • Sydes M.
        Practical methods for incorporating summary time-to-event data into meta-analysis.
        Trials. 2007; 8: 16
        • Duchateau L.
        • Collette L.
        • Sylvester R.
        • Pignon J.-P.
        Estimating number of events from the Kaplan-Meier curve for incorporation in a literature-based meta-analysis: what you don’t see you can’t get!.
        Biometrics. 2000; 56: 886-892
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Control Clin Trials. 1986; 7: 177-188
        • DerSimonian R.
        • Kacker R.
        • DerSimonian R.
        • Kacker R.
        Random-effects model for meta-analysis of clinical trials: an update.
        Contemp Clin Trials. 2007; 28: 105-114
        • Brockwell S.E.
        • Gordon I.R.
        A comparison of statistical methods for meta-analysis.
        Stat Med. 2001; 20: 825-840
        • Smith T.C.
        • Spiegelhalter D.J.
        • Thomas A.
        Bayesian approaches to random-effects meta-analysis: a comparative study.
        Stat Med. 1995; 14: 2685-2699
        • Sweeting M.J.
        • Sutton A.J.
        • Lambert P.C.
        What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
        Stat Med. 2004; 23: 1351-1375
        • Mehta C.R.
        The exact analysis of contingency tables in medical research.
        Cancer Treat Res. 1995; 75: 177-202
        • Mehta C.R.
        • Patel N.R.
        Exact logistic regression: theory and examples.
        Stat Med. 1995; 14: 2143-2160
        • Friedrich J.O.
        • Adhikari N.K.
        • Beyene J.
        • Friedrich J.O.
        • Adhikari N.K.J.
        • Beyene J.
        Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data.
        BMC Med Res Methodol. 2007; 7: 5
        • Sankey S.
        • Weissfeld L.
        • Fine M.
        • Kapoor W.
        An assessment of the use of the continuity correction for sparse data in meta-analysis.
        Comm Stat Simulat Comput. 1996; 25: 1031-1056
        • Hamza T.H.
        • van Houwelingen H.C.
        • Stijnen T.
        • Hamza T.H.
        • van Houwelingen H.C.
        • Stijnen T.
        The binomial distribution of meta-analysis was preferred to model within-study variability.
        J Clin Epidemiol. 2008; 61: 41-51
        • Lambert P.C.
        • Sutton A.J.
        • Burton P.R.
        • Abrams K.R.
        • Jones D.R.
        • Lambert P.C.
        • et al.
        How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS.
        Stat Med. 2005; 24: 2401-2428
        • The BUGS Project
        WinBUGS.
        (Available at) (Accessed May 31, 2010)
        • Lau J.
        • Schmid C.H.
        • Chalmers T.C.
        Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.
        J Clin Epidemiol. 1995; 48 (discussion 9–60): 45-57
        • Higgins J.P.
        • Thompson S.G.
        • Deeks J.J.
        • Altman D.G.
        Measuring inconsistency in meta-analyses.
        BMJ. 2003; 327: 557-560
        • Ioannidis J.P.
        • Patsopoulos N.A.
        • Evangelou E.
        • Ioannidis J.P.A.
        • Patsopoulos N.A.
        • Evangelou E.
        Uncertainty in heterogeneity estimates in meta-analyses.
        BMJ. 2007; 335: 914-916
        • Lau J.
        • Ioannidis J.P.
        • Schmid C.H.
        Summing up evidence: one answer is not always enough.
        Lancet. 1998; 351: 123-127
        • Schmid C.H.
        • Lau J.
        • McIntosh M.W.
        • Cappelleri J.C.
        An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials.
        Stat Med. 1998; 17: 1923-1942
        • Schmid C.H.
        • Stark P.C.
        • Berlin J.A.
        • Landais P.
        • Lau J.
        Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors.
        J Clin Epidemiol. 2004; 57: 683-697
        • Higgins J.P.
        • Thompson S.G.
        • Higgins J.P.T.
        • Thompson S.G.
        Controlling the risk of spurious findings from meta-regression.
        Stat Med. 2004; 23: 1663-1682
        • Slutsky J.
        • Atkins D.
        • Chang S.
        • Sharp B.
        Comparing medical interventions: AHRQ and the effective health-care program.
        J Clin Epidemiol. 2010; 63: 481-483
        • Concato J.
        • Shah N.
        • Horwitz R.I.
        Randomized, controlled trials, observational studies, and the hierarchy of research designs.
        New Engl J Med. 2000; 342: 1887-1892
        • Benson K.
        • Hartz A.J.
        A comparison of observational studies and randomized, controlled trials.
        New Engl J Med. 2000; 342: 1878-1886
        • Ioannidis J.P.A.
        • Haidich A.B.
        • Pappa M.
        • Pantazis N.
        • Kokori S.I.
        • Tektonidou M.G.
        • et al.
        Comparison of evidence of treatment effects in randomized and nonrandomized studies.
        JAMA. 2001; 286: 821-830
        • Ioannidis J.P.
        • Ioannidis J.P.A.
        Contradicted and initially stronger effects in highly cited clinical research.
        JAMA. 2005; 294: 218-228
        • Droitcour J.
        • Silberman G.
        • Chelimsky E.
        A new form of meta-analysis for combining results from randomized clinical trials and medical-practice databases.
        Int J Technol Assess Health Care. 1993; 9: 440-449
        • Prevost T.C.
        • Abrams K.R.
        • Jones D.R.
        Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening.
        Stat Med. 2000; 19: 3359-3376
        • Olkin I.
        Re: “A critical look at some popular meta-analytic methods”.
        Am J Epidemiol. 1994; 140 (discussion 300–301): 297-299