Original Article| Volume 114, P84-94, October 2019

Rapid network meta-analysis using data from Food and Drug Administration approval packages is feasible but with limitations



      To test rapid approaches that use [email protected] (a public database of approved drugs) and to identify trials and to compare these two sources with bibliographic databases as an evidence base for a systematic review and network meta-analysis (NMA).

      Study Design and Setting

      We searched bibliographic databases, [email protected], and for eligible trials on first-line glaucoma medications. We extracted data, assessed risk of bias, and examined the completeness and consistency of information provided by different sources. We fitted random-effects NMA models separately for trials identified from each source and for all unique trials from three sources.


      We identified 138 unique trials including 29,394 participants on 15 first-line glaucoma medications. For a given trial, information reported was sometimes inconsistent across data sources. Journal articles provided the most information needed for a systematic review; trial registrations provided the least. Compared to an NMA including all unique trials, we were able to generate reasonably precise effect estimates and similar relative rankings for available interventions using trials from [email protected] alone (but not


      A rapid NMA approach using data from [email protected] is feasible but has its own limitations. Reporting of trial design and results can be improved in both the drug approval packages and on


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Higgins J.P.T.
        GSe. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011].
        The Cochrane Collaboration, 2011 (Available at)
        • Shojania K.G.
        • Sampson M.
        • Ansari M.T.
        • Ji J.
        • Doucette S.
        • Moher D.
        How quickly do systematic reviews go out of date? A survival analysis.
        Ann Intern Med. 2007; 147: 224-233
        • Sampson M.
        • Shojania K.G.
        • Garritty C.
        • Horsley T.
        • Ocampo M.
        • Moher D.
        Systematic reviews can be produced and published faster.
        J Clin Epidemiol. 2008; 61: 531-536
        • Beller E.M.
        • Chen J.K.
        • Wang U.L.
        • Glasziou P.P.
        Are systematic reviews up-to-date at the time of publication?.
        Syst Rev. 2013; 2: 36
        • Tsafnat G.
        • Glasziou P.
        • Choong M.K.
        • Dunn A.
        • Galgani F.
        • Coiera E.
        Systematic review automation technologies.
        Syst Rev. 2014; 3: 74
        • Hartling L.
        • Guise J.M.
        • Kato E.
        • Anderson J.
        • Aronson N.
        • Belinson S.
        • et al.
        AHRQ comparative effectiveness reviews.
        in: EPC methods: an exploration of methods and context for the production of rapid reviews. Agency for Healthcare Research and Quality (US), Rockville, MD2015
        • Ganann R.
        • Ciliska D.
        • Thomas H.
        Expediting systematic reviews: methods and implications of rapid reviews.
        Implement Sci. 2010; 5: 56
        • Khangura S.
        • Konnyu K.
        • Cushman R.
        • Grimshaw J.
        • Moher D.
        Evidence summaries: the evolution of a rapid review approach.
        Syst Rev. 2012; 1: 10
        • AHRQ Methods for Effective Health Care
        Methods guide for effectiveness and comparative effectiveness reviews.
        Agency for Healthcare Research and Quality (US), Rockville, MD2008
        • Halladay C.W.
        • Trikalinos T.A.
        • Schmid I.T.
        • Schmid C.H.
        • Dahabreh I.J.
        Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions.
        J Clin Epidemiol. 2015; 68: 1076-1084
        • Mayo-Wilson E.
        • Li T.
        • Fusco N.
        • Dickersin K.
        Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study).
        Res Synth Methods. 2018; 9: 2-12
        • Institute of Medicine
        Finding what works in health care: standards for systematic reviews.
        The National Academies Press, Washington, FC2011
        • Page M.J.
        • Shamseer L.
        • Altman D.G.
        • Tetzlaff J.
        • Sampson M.
        • Tricco A.C.
        • et al.
        Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.
        PLoS Med. 2016; 13: e1002028
        • Rosman L.T.C.
        • Li M.
        • Li T.
        • Saldanha I.
        • Dickersin K.
        Teaching searching in an intensive systematic review course: "how many citations should I expect to review?".
        Cochrane Colloquium, Quebec, Canada2013
        • Wallace B.C.
        • Trikalinos T.A.
        • Lau J.
        • Brodley C.
        • Schmid C.H.
        Semi-automated screening of biomedical citations for systematic reviews.
        BMC Bioinformatics. 2010; 11: 55
        • Chalmers I.
        The cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care.
        Ann N Y Acad Sci. 1993; 703 ([discussion 63-65]): 156-163
        • Garg A.X.
        • Hackam D.
        • Tonelli M.
        Systematic review and meta-analysis: when one study is just not enough.
        Clin J Am Soc Nephrol. 2008; 3: 253-260
        • Boland M.V.
        • Ervin A.M.
        • Friedman D.
        • Jampel H.
        • Hawkins B.
        • Volenweider D.
        • et al.
        AHRQ comparative effectiveness reviews.
        in: Treatment for glaucoma: comparative effectiveness. Agency for Healthcare Research and Quality (US), Rockville, MD2012
        • Boland M.V.
        • Ervin A.M.
        • Friedman D.S.
        • Jampel H.D.
        • Hawkins B.S.
        • Vollenweider D.
        • et al.
        Comparative effectiveness of treatments for open-angle glaucoma: a systematic review for the U.S. Preventive Services Task Force.
        Ann Intern Med. 2013; 158: 271-279
        • Li T.
        • Lindsley K.
        • Rouse B.
        • Hong H.
        • Shi Q.
        • Friedman D.S.
        • et al.
        Comparative effectiveness of first-line medications for primary open-angle glaucoma: a systematic review and network meta-analysis.
        Ophthalmology. 2016; 123: 129-140
        • Li T.
        • Puhan M.A.
        • Vedula S.S.
        • Singh S.
        • Dickersin K.
        Network meta-analysis-highly attractive but more methodological research is needed.
        BMC Med. 2011; 9: 79
        • Salanti G.
        • Ades A.E.
        • Ioannidis J.P.
        Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial.
        J Clin Epidemiol. 2011; 64: 163-171
        • Tonin F.S.
        • Rotta I.
        • Mendes A.M.
        • Pontarolo R.
        Network meta-analysis: a technique to gather evidence from direct and indirect comparisons.
        Pharm Pract. 2017; 15: 943
        • Rouse B.
        • Chaimani A.
        • Li T.
        Network meta-analysis: an introduction for clinicians.
        Intern Emerg Med. 2017; 12: 103-111
        • Hartling L.
        • Guise J.M.
        • Hempel S.
        • Featherstone R.
        • Mitchell M.D.
        • Motu’apuaka M.L.
        • et al.
        Fit for purpose: perspectives on rapid reviews from end-user interviews.
        Syst Rev. 2017; 6: 32
        • Hopewell S.
        • McDonald S.
        • Clarke M.
        • Egger M.
        Grey literature in meta-analyses of randomized trials of health care interventions.
        Cochrane Database Syst Rev. 2007; : Mr000010
        • Vandvik P.O.
        • Brignardello-Petersen R.
        • Guyatt G.H.
        Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews?.
        BMC Med. 2016; 14: 59
      1. NIH policy on the dissemination of NIH-funded clinical trial information.
        in: Health NIo. National Institutes of Health, Bethesda, MD2016: 64922-64928
        • Hudson K.L.
        • Lauer M.S.
        • Collins F.S.
        Toward a new era of trust and transparency in clinical trials.
        JAMA. 2016; 316: 1353-1354
        • Zarin D.A.
        • Tse T.
        • Williams R.J.
        • Carr S.
        Trial reporting in - the final rule.
        N Engl J Med. 2016; 375: 1998-2004
        • Anderson M.L.
        • Chiswell K.
        • Peterson E.D.
        • Tasneem A.
        • Topping J.
        • Califf R.M.
        Compliance with results reporting at
        N Engl J Med. 2015; 372: 1031-1039
        • Tsertsvadze A.
        • Chen Y.F.
        • Moher D.
        • Sutcliffe P.
        • McCarthy N.
        How to conduct systematic reviews more expeditiously?.
        Syst Rev. 2015; 4: 160
        • Trinquart L.
        • Abbe A.
        • Ravaud P.
        Impact of reporting bias in network meta-analysis of antidepressant placebo-controlled trials.
        PLoS One. 2012; 7: e35219
        • Cameron C.
        Enhancing the use of network meta-analysis to synthesize information on benefits and harms of drugs to support regulatory and reimbursement decisions in Canada: School of Epidemiology.
        Public Health and Preventive Medicine University of Ottawa, Ottawa, Canada2015
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Control Clin Trials. 1986; 7: 177-188
        • Harris R.
        • Bradburn M.
        • Deeks J.
        • Harbord R.
        • Altman D.
        • Sterne J.
        Metan: fixed- and random-effects meta-analysis.
        Stata J. 2008; 8: 3-28
        • White I.R.
        Multivariate random-effects meta-regression: updates to mvmeta.
        Stata J. 2011; 11: 255-270
        • Chaimani A.
        • Higgins J.P.
        • Mavridis D.
        • Spyridonos P.
        • Salanti G.
        Graphical tools for network meta-analysis in STATA.
        PLoS One. 2013; 8: e76654
        • Chaimani A.
        • Salanti G.
        Visualizing assumptions and results in network meta-analysis: the network graphs package.
        Stata J. 2015; 15: 905-950
        • White I.R.
        Network meta-analysis.
        Stata J. 2015; 15: 951-985
        • Salanti G.
        Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool.
        Res Synth Methods. 2012; 3: 80-97
        • Jansen J.P.
        • Naci H.
        Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers.
        BMC Med. 2013; 11: 159
        • Lumley T.
        Network meta-analysis for indirect treatment comparisons.
        Stat Med. 2002; 21: 2313-2324
        • Dias S.
        • Welton N.J.
        • Caldwell D.M.
        • Ades A.E.
        Checking consistency in mixed treatment comparison meta-analysis.
        Stat Med. 2010; 29: 932-944
        • Higgins J.P.
        • Jackson D.
        • Barrett J.K.
        • Lu G.
        • Ades A.E.
        • White I.R.
        Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies.
        Res Synth Methods. 2012; 3: 98-110
        • White I.R.
        • Barrett J.K.
        • Jackson D.
        • Higgins J.P.
        Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression.
        Res Synth Methods. 2012; 3: 111-125
        • Dias S.
        • Welton N.J.
        • Sutton A.J.
        • Caldwell D.M.
        • Lu G.
        • Ades A.E.
        Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials.
        Med Decis Making. 2013; 33: 641-656
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective publication of antidepressant trials and its influence on apparent efficacy.
        N Engl J Med. 2008; 358: 252-260
        • Hart B.
        • Lundh A.
        • Bero L.
        Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses.
        BMJ. 2012; 344: d7202
        • Roest A.M.
        • de Jonge P.
        • Williams C.D.
        • de Vries Y.A.
        • Schoevers R.A.
        • Turner E.H.
        Reporting bias in clinical trials investigating the efficacy of second-generation antidepressants in the treatment of anxiety disorders: a report of 2 meta-analyses.
        JAMA Psychiatry. 2015; 72: 500-510
        • Turner E.H.
        How to access and process FDA drug approval packages for use in research.
        BMJ. 2013; 347: f5992
        • Mathieu S.
        • Boutron I.
        • Moher D.
        • Altman D.G.
        • Ravaud P.
        Comparison of registered and published primary outcomes in randomized controlled trials.
        JAMA. 2009; 302: 977-984
        • Dwan K.
        • Altman D.G.
        • Cresswell L.
        • Blundell M.
        • Gamble C.L.
        • Williamson P.R.
        Comparison of protocols and registry entries to published reports for randomised controlled trials.
        Cochrane Database Syst Rev. 2011; : Mr000031
        • Scherer R.W.
        • Huynh L.
        • Ervin A.M.
        • Taylor J.
        • Dickersin K. registration can supplement information in abstracts for systematic reviews: a comparison study.
        BMC Med Res Methodol. 2013; 13: 79
        • Becker J.E.
        • Krumholz H.M.
        • Ben-Josef G.
        • Ross J.S.
        Reporting of results in and high-impact journals.
        JAMA. 2014; 311: 1063-1065
        • Hartung D.M.
        • Zarin D.A.
        • Guise J.M.
        • McDonagh M.
        • Paynter R.
        • Helfand M.
        Reporting discrepancies between the results database and peer-reviewed publications.
        Ann Intern Med. 2014; 160: 477-483
        • Halfpenny N.J.
        • Quigley J.M.
        • Thompson J.C.
        • Scott D.A.
        Value and usability of unpublished data sources for systematic reviews and network meta-analyses.
        Evid Based Med. 2016; 21: 208-213
        • Bennett D.A.
        • Jull A.
        FDA: untapped source of unpublished trials.
        Lancet. 2003; 361: 1402-1403
        • Schulz K.F.
        • Altman D.G.
        • Moher D.
        CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.
        BMJ. 2010; 340: c332
        • Li T.
        What’s not shared—building on the FDA’s transparency momentum.
        BMJ. 2018; (Available at)
        • Marshall I.J.
        • Marshall R.
        • Wallace B.C.
        • Brassey J.
        • Thomas J.
        Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study.
        J Clin Epidemiol. 2019; 109: 30-41