Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review



      The aim of the article was to identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews.

      Study Design and Setting

      A systematic review was conducted, searching MEDLINE, EMBASE, and the Cochrane Library from inception to September 1, 2016. Quality appraisal of included studies was undertaken using a modified Quality Assessment of Diagnostic Accuracy Studies 2, and key results on accuracy, reliability, efficiency of a methodology, or impact on results and conclusions were extracted.


      After screening 5,600 titles and abstracts and 245 full-text articles, 37 studies were included. For screening, studies supported the involvement of two independent experienced reviewers and the use of Google Translate when screening non-English articles. For data abstraction, studies supported involvement of experienced reviewers (especially for continuous outcomes) and two independent reviewers, use of dual monitors, graphical data extraction software, and contacting authors. For quality appraisal, studies supported intensive training, piloting quality assessment tools, providing decision rules for poorly reported studies, contacting authors, and using structured tools if different study designs are included.


      Few studies exist documenting common systematic review practices. Included studies support several systematic review practices. These results provide an updated evidence-base for current knowledge synthesis guidelines and methods requiring further research.


      To read this article in full you will need to make a payment


        • Moher D.
        • Shamseer L.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
        Syst Rev. 2015; 4: 1
        • Higgins J.
        • Green S.
        Cochrane handbook for systematic reviews of interventions version 5.1.0.
        The Cochrane Collaboration, 2011 (Available at)
      1. Systematic reviews: CRD's guidance for undertaking reviews in health care. University of York, Centre for Reviews & Dissemination, 2009 (Available at)
        • Owens D.
        • Lohr K.N.
        • Atkins D.
        • Treadwell J.
        • Reston J.
        • Bass E.
        • et al.
        Methods guide for effectiveness and comparative effectiveness reviews.
        Agency for Healthcare Research and Quality, Rockville, MD2014
      2. Institute of medicine committee on standards for systematic reviews of comparative effectiveness R.
        in: Eden J. Levit L. Berg A. Morton S. Finding what works in health care: standards for systematic reviews. National Academies Press (US) Copyright 2011 by the National Academy of Sciences. All rights reserved., Washington (DC)2011
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        Ann Intern Med. 2009; 4: 264-269
      3. Joanna Briggs Institute Reviewers’ Manual: 2015 edition/supplement. Methodology for JBI Scoping Reviews, Australia2015
      4. Methodological expectations of Campbell Collaboration intervention reviews: conduct standards. 2017 (Available at)
        • McGowan J.
        • Sampson M.
        • Salzwedel D.M.
        • Cogo E.
        • Foerster V.
        • Lefebvre C.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        J Clin Epidemiol. 2016; 75: 40-46
        • Higgins J.P.
        • Altman D.G.
        • Gotzsche P.C.
        • Juni P.
        • Moher D.
        • Oxman A.D.
        • et al.
        The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
      5. Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. The Ottawa Hospital Research Institute: Ottawa, ON.

        • Whiting P.
        • Rutjes A.W.
        • Reitsma J.B.
        • Bossuyt P.M.
        • Kleijnen J.
        The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews.
        BMC Med Res Methodol. 2003; 3: 25
        • Lucas N.P.
        • Macaskill P.
        • Irwig L.
        • Bogduk N.
        The development of a quality appraisal tool for studies of diagnostic reliability (QAREL).
        J Clin Epidemiol. 2010; 63: 854-861
        • Moher D.
        • Hopewell S.
        • Schulz K.F.
        • Montori V.
        • Gotzsche P.C.
        • Devereaux P.J.
        • et al.
        CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials.
        Int J Surg. 2012; 10: 28-55
        • Vandenbroucke J.P.
        • Elm E.
        • Altman D.G.
        • Gøtzsche P.C.
        • Mulrow C.D.
        • Pocock S.J.
        • et al.
        Strengthening the reporting of observational studies in epidemiology (strobe): explanation and elaboration.
        Ann Intern Med. 2007; 147: W163-W194
        • Mathes T.
        • Klassen P.
        • Pieper D.
        Frequency of data extraction errors and methods to increase data extraction quality: a methodological review.
        BMC Med Res Methodol. 2017; 17: 152
        • Tricco A.C.
        • Antony J.
        • Zarin W.
        • Strifler L.
        • Ghassemi M.
        • Ivory J.
        • et al.
        A scoping review of rapid review methods.
        BMC Med. 2015; 13: 224
        • Tricco A.
        • Robson R.
        • Thomas S.
        • Pham B.
        • Page M.
        Accuracy, reliability, impact, and efficiency of different methods for selecting studies, abstracting data, and appraising quality in a systematic review: a systematic review protocol.
        PROSPERO, 2016 (Available at)
        • Moher D.
        • Stewart L.
        • Shekelle P.
        All in the family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more.
        Syst Rev. 2015; 4: 183
        • Wallace B.C.
        • Dahabreh I.J.
        • Schmid C.H.
        • Lau J.
        • Trikalinos T.A.
        Modernizing the systematic review process to inform comparative effectiveness: tools and methods.
        J Comp Eff Res. 2013; 2: 273-282
        • O'Mara-Eves A.
        • Thomas J.
        • McNaught J.
        • Miwa M.
        • Ananiadou S.
        Erratum to: using text mining for study identification in systematic reviews: a systematic review of current approaches.
        Syst Rev. 2015; 4: 59
        • Tsafnat G.
        • Dunn A.
        • Glasziou P.
        • Coiera E.
        The automation of systematic reviews.
        BMJ. 2013; 346: f139
      6. Newton D. Synthesi.SR. Toronto, Ontario: Knowledge Translation Program. Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, ON2012
        • Edwards P.
        • Clarke M.
        • DiGuiseppi C.
        • Pratap S.
        • Roberts I.
        • Wentz R.
        Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records.
        Stat Med. 2002; 21: 1635-1640
        • Doust J.A.
        • Pietrzak E.
        • Sanders S.
        • Glasziou P.P.
        Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract.
        J Clin Epidemiol. 2005; 58: 444-449
        • Yip R.
        • Islami F.
        • Zhao S.
        • Tao M.
        • Yankelevitz D.F.
        • Boffetta P.
        Errors in systematic reviews: an example of computed tomography screening for lung cancer.
        Eur J Cancer Prev. 2014; 23: 43-48
        • Shemilt I.
        • Khan N.
        • Park S.
        • Thomas J.
        Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
        Syst Rev. 2016; 5: 140
        • Ng L.
        • Pitt V.
        • Huckvale K.
        • Clavisi O.
        • Turner T.
        • Gruen R.
        • et al.
        Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students.
        Syst Rev. 2014; 3: 121
        • Cooper M.
        • Ungar W.
        • Zlotkin S.
        An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines.
        Public Health Nutr. 2006; 9: 494-500
        • Freitas de Souza R.
        • Sequeira P.
        • Nasser M.
        Is Google Translate useful for the selection of studies to be included in Cochrane reviews.
        (17th Cochrane Colloquium, Singapore)2009: 11-14
        • Busse J.W.
        • Bruno P.
        • Malik K.
        • Connell G.
        • Torrance D.
        • Ngo T.
        • et al.
        An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review.
        J Clin Epidemiol. 2014; 67: 547-553
        • Berlin J.A.
        • On behalf of University of Pennsylvania Meta-analysis Blinding Study Group
        Does blinding of readers affect the results of meta-analyses?.
        Lancet. 1997; 350: 185-186
        • Mateen F.J.
        • Oh J.
        • Tergas A.I.
        • Bhayani N.H.
        • Kamdar B.B.
        Titles versus titles and abstracts for initial screening of articles for systematic reviews.
        J Clin Epidemiol. 2013; 5: 89-95
        • Wang Z.
        • Asi N.
        • Elraiyah T.A.
        • Abu Dabrh A.M.
        • Undavalli C.
        • Glasziou P.
        • et al.
        Dual computer monitors to increase efficiency of conducting systematic reviews.
        J Clin Epidemiol. 2014; 67: 1353-1357
        • Buscemi N.
        • Hartling L.
        • Vandermeer B.
        • Tjosvold L.
        • Klassen T.P.
        Single data extraction generated more errors than double data extraction in systematic reviews.
        J Clin Epidemiol. 2006; 59: 697-703
        • Horton J.
        • Vandermeer B.
        • Hartling L.
        • Tjosvold L.
        • Klassen T.P.
        • Buscemi N.
        Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.
        J Clin Epidemiol. 2010; 63: 289-298
        • Tendal B.
        • Higgins J.P.
        • Juni P.
        • Hrobjartsson A.
        • Trelle S.
        • Nuesch E.
        • et al.
        Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study.
        BMJ. 2009; 339: b3128
        • Jayaram M.K.
        • Mansi K.
        • Haynes E.
        • Adams C.E.
        • Furtado V.A.
        Catch 22: is the future of systematic reviewing only for the experienced? [unknown].
        Cochrane Schizophrenia Group, 2014
      7. Florence Z, Schulz T, Pearson A. Inter-reviewer agreement: an analysis of the degree to which agreement occurs when using tools for the appraisal, extraction and meta-synthesis of qualitative research findings. Abstracts of the 13th Cochrane Colloquium. Melbourne, Australia 2005. p. 69.

        • Gresham G.
        • Matsumura S.
        • Li T.
        Faster may not be better: data abstraction for systematic reviews.
        Cochrane Eyes and Vision Group. US Cochrane Center, U. S.2014
        • Balk E.M.
        • Chung M.
        • Chen M.L.
        • Trikalinos T.A.
        • Kong Win Chang L.
        Assessing the accuracy of Google Translate to allow data extraction from trials published in non-English languages. Methods research report.
        Agency for Healthcare Research and Quality (US), Rockville (MD)2013
        • Selph S.S.
        • Ginsburg A.D.
        • Chou R.
        Impact of contacting study authors to obtain additional data for systematic reviews: diagnostic accuracy studies for hepatic fibrosis.
        Syst Rev. 2014; 3: 107
        • Gibson C.A.
        • Bailey B.W.
        • Carper M.J.
        • LeCheminant J.D.
        • Kirk E.P.
        • Huang G.
        • et al.
        Author contacts for retrieval of data for a meta-analysis on exercise and diet restriction.
        Int J Technol Assess Health Care. 2006; 22: 267-270
      8. Cahill K, Perera R, Selwood M. Electronic extraction of graphical data [abstract]. XV Cochrane Colloquium, Sao Paulo, Brazil 2007, 153-154p 2007 Oct 23-27.

        • de Oliveira I.R.
        • Santos-Jesus R.
        • Po A.L.
        • Poolsup N.
        Extracting numerical data from published reports of pharmacokinetics investigations: method description and validation.
        Fundam Clin Pharmacol. 2003; 17: 471-472
      9. Higgins J.P.T.G.S. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. John Wiley & Sons, Ltd., Chichester, UK2006
        Date accessed: September , 2006
        • da Costa B.R.
        • Beckett B.
        • Diaz A.
        • Resta N.M.
        • Johnston B.C.
        • Egger M.
        • et al.
        Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study.
        Syst Rev. 2017; 6: 44
        • Sands M.L.
        • Murphy J.R.
        Use of kappa statistic in determining validity of quality filtering for meta-analysis: a case study of the health effects of electromagnetic radiation.
        J Clin Epidemiol. 1996; 49: 1045-1051
        • Fourcade L.
        • Boutron I.
        • Moher D.
        • Ronceray L.
        • Baron G.
        • Ravaud P.
        Development and evaluation of a pedagogical tool to improve understanding of a quality checklist: a randomised controlled trial.
        PLoS Clin Trials. 2007; 2
        • Oremus M.
        • Oremus C.
        • Hall G.B.C.
        • McKinnon M.C.
        • Graham A.
        • Gregory C.
        • et al.
        Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales.
        BMJ Open. 2012; 2
        • Jadad A.R.
        • Moore R.A.
        • Carroll D.
        • Jenkinson C.
        • Reynolds D.J.
        • Gavaghan D.J.
        • et al.
        Assessing the quality of reports of randomized clinical trials: is blinding necessary?.
        Control Clin Trials. 1996; 17: 1-12
        • Berard A.
        • Andreu N.
        • Tetrault J.
        • Niyonsenga T.
        • Myhal D.
        Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis.
        Ann Epidemiol. 2000; 10: 498-503
        • Clark H.D.
        • Wells G.A.
        • Huet C.
        • McAlister F.A.
        • Salmi L.R.
        • Fergusson D.
        • et al.
        Assessing the quality of randomized trials: reliability of the Jadad scale.
        Control Clin Trials. 1999; 20: 448-452
        • Verhagen A.P.
        • de Vet H.C.
        • de Bie R.A.
        • Kessels A.G.
        • Boers M.
        • Knipschild P.G.
        Balneotherapy and quality assessment: interobserver reliability of the Maastricht criteria list and the need for blinded quality assessment.
        J Clin Epidemiol. 1998; 51: 335-341
        • Armijo-Olivo S.
        • Ospina M.
        • da Costa B.R.
        • Egger M.
        • Saltaji H.
        • Fuentes J.
        • et al.
        Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials.
        PLoS One. 2014; 9: e96920
        • Hartling L.
        • Hamm M.P.
        • Milne A.
        • Vandermeer B.
        • Santaguida P.L.
        • Ansari M.
        • et al.
        Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs.
        J Clin Epidemiol. 2013; 66: 973-981
        • Robertson C.
        • Ramsay C.
        • Gurung T.
        • Mowatt G.
        • Pickard R.
        • Sharma P.
        • et al.
        Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non-randomised study designs applied in a health technology assessment setting.
        Res Synth Methods. 2014; 5: 200-211
        • Vale C.L.
        • Tierney J.F.
        • Burdett S.
        Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews.
        BMJ. 2013; 346: f1798
        • Littlewood C.
        • Ashton J.
        • Chance-Larsen K.
        • May M.
        • Sturrock B.
        The quality of reporting might not reflect the quality of the study: implications for undertaking and appraising a systematic review.
        J Man Manip Ther. 2012; 20: 130-134
        • Crowe M.
        • Sheppard L.
        • Campbell A.
        Comparison of the effects of using the Crowe Critical Appraisal Tool versus informal appraisal in assessing health research: a randomised trial.
        Int J Evid Based Healthc. 2011; 9: 444-449
        • Dixon-Woods M.
        • Sutton A.
        • Shaw R.
        • Miller T.
        • Smith J.
        • Young B.
        • et al.
        Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods.
        J Health Serv Res Policy. 2007; 12: 42-47