Original Article| Volume 68, ISSUE 6, P617-626, June 2015

Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews

Published:February 05, 2015DOI:



      To determine whether librarian and information specialist authorship was associated with better reported systematic review (SR) search quality.

      Study Design and Setting

      SRs from high-impact general internal medicine journals were reviewed for search quality characteristics and reporting quality by independent reviewers using three instruments, including a checklist of Institute of Medicine Recommended Standards for the Search Process and a scored modification of the Peer Review of Electronic Search Strategies instrument.


      The level of librarian and information specialist participation was significantly associated with search reproducibility from reported search strategies (Χ2 = 23.5; P < 0.0001). Librarian co-authored SRs had significantly higher odds of meeting 8 of 13 analyzed search standards than those with no librarian participation and six more than those with mentioned librarian participation. One-way ANOVA showed that differences in total search quality scores between all three groups were statistically significant (F2,267 = 10.1233; P < 0.0001).


      Problems remain with SR search quality and reporting. SRs with librarian or information specialist co-authors are correlated with significantly higher quality reported search strategies. To minimize bias in SRs, authors and editors could encourage librarian engagement in SRs including authorship as a potential way to help improve documentation of the search strategy.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Guyatt G.
        Evidence-based health care: a look into the future. EBHC Conference 2013; 30 October.
        (Taormina, Italy)2013 (Available at) (. Accessed December 1, 2013)
        • Bastian H.
        • Glasziou P.
        • Chalmers I.
        Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?.
        PLoS Med. 2010; 7: e1000326
        • Moher D.
        • Tetzlaff J.
        • Tricco A.C.
        • Sampson M.
        • Altman D.G.
        Epidemiology and reporting characteristics of systematic reviews.
        PLoS Med. 2007; 4: e78
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6: e1000097
        • Moher D.
        • Cook D.J.
        • Eastwood S.
        • Olkin I.
        • Rennie D.
        • Stroup D.F.
        Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses.
        Lancet. 1999; 354: 1896-1900
        • Stroup D.F.
        • Berlin J.A.
        • Morton S.C.
        • Olkin I.
        • Williamson G.D.
        • Rennie D.
        • et al.
        Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) group.
        JAMA. 2000; 283: 2008-2012
        • Shea B.J.
        • Grimshaw J.M.
        • Wells G.A.
        • Boers M.
        • Andersson N.
        • Hamel C.
        • et al.
        Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
        BMC Med Res Methodol. 2007; 7: 10
        • Institute of Medicine (U.S.)
        Committee on standards for systematic reviews of comparative effectiveness research.
        in: Eden J. Finding what works in health care : standards for systematic reviews. National Academies Press, Washington, D.C2011 (xxii, 317 p. p)
        • Higgins J.P.T.
        • Green S.
        Cochrane handbook for systematic reviews of interventions.
        The Cochrane Collaboration, 2011 (Available at) ([updated March]; Accessed December 18, 2013)
        • Yoshii A.
        • Plaut D.A.
        • McGraw K.A.
        • Anderson M.J.
        • Wellik K.E.
        Analysis of the reporting of search strategies in Cochrane systematic reviews.
        J Med Libr Assoc. 2009; 97: 21-29
        • Sampson M.
        • McGowan J.
        • Tetzlaff J.
        • Cogo E.
        • Moher D.
        No consensus exists on search reporting methods for systematic reviews.
        J Clin Epidemiol. 2008; 61: 748-754
        • Sampson M.
        • McGowan J.
        Errors in search strategies were identified by type and frequency.
        J Clin Epidemiol. 2006; 59: 1057-1063
        • Golder S.
        • Loke Y.
        • McIntosh H.M.
        Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects.
        J Clin Epidemiol. 2008; 61: 440-448
        • Maggio L.A.
        • Tannery N.H.
        • Kanter S.L.
        Reproducibility of literature search reporting in medical education reviews.
        Acad Med. 2011; 86: 1049-1054
        • Faggion Jr., C.M.
        • Atieh M.A.
        • Park S.
        Search strategies in systematic reviews in periodontology and implant dentistry.
        J Clin Periodontol. 2013; 40: 883-888
        • Golder S.
        • Loke Y.K.
        • Zorzela L.
        Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011.
        J Clin Epidemiol. 2013; 66: 253-260
        • McGowan J.
        • Sampson M.
        Systematic reviews need systematic searchers.
        J Med Libr Assoc. 2005; 93: 74-80
        • Weller A.C.
        Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports.
        J Med Libr Assoc. 2004; 92: 163-164
        • Lefebvre C.
        • Glanville J.
        • Wieland L.S.
        • Coles B.
        • Weightman A.L.
        Methodological developments in searching for studies for systematic reviews: past, present and future?.
        Syst Rev. 2013; 2: 78
        • Patrick T.B.
        • Demiris G.
        • Folk L.C.
        • Moxley D.E.
        • Mitchell J.A.
        • Tao D.
        Evidence-based retrieval in evidence-based medicine.
        J Med Libr Assoc. 2004; 92: 196-199
        • Zhang L.
        • Sampson M.
        • McGowan J.
        Reporting of the role of the expert searcher in Cochrane Reviews.
        Evid Based Libr Inf Pract. 2006; 1: 3-16
        • Li L.
        • Tian J.
        • Tian H.
        • Moher D.
        • Liang F.
        • Jiang T.
        • et al.
        Network meta-analyses could be improved by searching more sources and by involving a librarian.
        J Clin Epidemiol. 2014; 67: 1001-1007
        • McKibbon K.A.
        • Haynes R.B.
        • Dilks C.J.
        • Ramsden M.F.
        • Ryan N.C.
        • Baker L.
        • et al.
        How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches.
        Comput Biomed Res. 1990; 23: 583-593
        • Hausner E.
        • Waffenschmidt S.
        • Kaiser T.
        • Simon M.
        Routine development of objectively derived search strategies.
        Syst Rev. 2012; 1: 19
        • Erwin P.J.
        By the clock: how much time does an expert search take?.
        MLA News. 2004; 1: 12
        • Greenhalgh T.
        • Peacock R.
        Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources.
        BMJ. 2005; 331: 1064-1065
      1. Journal citation reports: Thomson Reuters. 2011 ([cited 2012]. Available at) (. Accessed August 3, 2012)
        • Montori V.M.
        • Wilczynski N.L.
        • Morgan D.
        • Haynes R.B.
        Systematic reviews: a cross-sectional study of location and citation counts.
        BMC Med. 2003; 1: 2
      2. Search strategy used to create the systematic reviews subset on PubMed. National Library of Medicine, Bethesda, MD2013 (. Available at) (. [updated 26 February]; Accessed December 13, 2013)
        • Montori V.M.
        • Wilczynski N.L.
        • Morgan D.
        • Haynes R.B.
        Optimal search strategies for retrieving systematic reviews from Medline: analytical survey.
        BMJ. 2005; 330: 68
        • Shojania K.G.
        • Bero L.A.
        Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy.
        Eff Clin Pract. 2001; 4: 157-162
        • Nelson P.
        • Wludyka P.
        • Copeland K.
        The analysis of means: a graphical method for comparing means, rates, and proportions.
        Society for Industrial and Applied Mathematics, Philadelphia, PA2005
        • Hayes A.F.
        • Krippendorff K.
        Answering the call for a standard reliability measure for coding data.
        Commun Methods Meas. 2007; 1: 77-89
        • Hayes A.F.
        KALPHA (SAS Version).
        2011 ([updated 12 June; cited 2012]. Available at) (. Accessed March 23, 2013)
        • Sampson M.
        • McGowan J.
        • Cogo E.
        • Grimshaw J.
        • Moher D.
        • Lefebvre C.
        An evidence-based practice guideline for the peer review of electronic search strategies.
        J Clin Epidemiol. 2009; 62: 944-952
        • Sampson M.
        • McGowan J.
        • Lefebvre C.
        • Moher D.
        • Grimshaw J.M.
        PRESS: peer review of electronic search strategies.
        Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada2008 ([updated January] Available at) (. Accessed December 13, 2013)
        • Sampson M.
        • McGowan J.
        • Lefebvre C.
        • Moher D.
        • Grimshaw J.M.
        Appendix G: PRESS Checklist.
        Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada2008 (Available at) (. Accessed December 13, 2013)
        • De Swert K.
        Calculating inter-coder reliability in media content analysis using Krippendorff’s Alpha.
        2012 (Available at) (. [updated 2 January];Accessed December 13, 2013)
        • Karimi S.
        • Pohl S.
        • Scholer F.
        • Cavedon L.
        • Zobel J.
        Boolean versus ranked querying for biomedical systematic reviews.
        BMC Med Inform Decis Mak. 2010; 10: 58
        • National Center for Biotechnology Information
        How PubMed works: automatic term mapping.
        National Library of Medicine, Bethesda, MD2013 (Available at) (. [updated 17 December]; Accessed January 17, 2014)
        • Nahin A.M.
        PubMed now using the redesigned interface.
        NLM Tech Bull. 2009; : e22
        • Relevo R.
        • Paynter R.
        Peer review of search strategies.
        (Report No. 12-EHC068-EF. Methods Research Reports) Agency for Healthcare Research and Quality, Rockville, MD2012
        • Craven J.
        • Levay P.
        Recording database searches for systematic reviews—what is the value of adding a narrative to peer-review checklists? A case study of NICE interventional procedures guidance.
        Evid Based Libr Inf Pract. 2011; 6: 72-87
        • Li T.
        • Bartley G.B.
        Publishing systematic reviews in ophthalmology: new guidance for authors.
        Ophthalmology. 2014; 121: 438-439
        • Rethlefsen M.L.
        • Murad M.H.
        • Livingston E.H.
        Engaging medical librarians to improve the quality of review articles.
        JAMA. 2014; 312: 999-1000
        • Shea B.
        • Bouter L.M.
        • Grimshaw J.M.
        • Francis D.
        • Ortiz Z.
        • Wells G.A.
        • et al.
        Scope for improvement in the quality of reporting of systematic reviews. From the Cochrane Musculoskeletal Group.
        J Rheumatol. 2006; 33: 9-15
        • Flores-Mir C.
        • Major M.P.
        • Major P.W.
        Search and selection methodology of systematic reviews in orthodontics (2000-2004).
        Am J Orthod Dentofacial Orthop. 2006; 130: 214-217
        • Major M.P.
        • Major P.W.
        • Flores-Mir C.
        An evaluation of search and selection methods used in dental systematic reviews published in English.
        J Am Dent Assoc. 2006; 137: 1252-1257
      3. Methods guide for effectiveness and comparative effectiveness reviews. Agency for healthcare research and quality, Rockville MD2008 (Available at) (Accessed December 18, 2013)
        • Chandler J.
        • Churchill R.
        • Higgins J.
        • Lasserson T.
        • Tovey D.
        Methodological standards for the conduct of new Cochrane Intervention Reviews Version 2.3.
        The Cochrane Library, 2013 (Available at) (. [updated 2 December]; Accessed October 21, 2014)
        • Bramer W.M.
        • Giustini D.
        • Kramer B.M.
        • Anderson P.
        The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews.
        Syst Rev. 2013; 2: 115
        • Sampson M.
        • Tetzlaff J.
        • Urquhart C.
        Precision of healthcare systematic review searches in a cross-sectional sample.
        Res Synth Methods. 2011; 2: 119-125
        • Lau J.
        • Chang S.
        • Berkman N.
        • Ratichek S.J.
        • H B Brasure M.
        • et al.
        EPC response to IOM standards for systematic reviews. Research white paper.
        (Prepared by the Tufts Evidence-based Practice Center, Tufts Medical Center under Contract No. 290-2007-10055-I). AHRQ Publication No. 13-EHC006-EF Agency for Healthcare Research and Quality, Rockville, MD2013 (Available at) (. [updated April]; Accessed December 13, 2013)
        • Klimo Jr., P.
        • Thompson C.J.
        • Ragel B.T.
        • Boop F.A.
        Methodology and reporting of meta-analyses in the neurosurgical literature.
        J Neurosurg. 2014; 120: 796-810
        • Tunis A.S.
        • McInnes M.D.
        • Hanna R.
        • Esmail K.
        Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement?.
        Radiology. 2013; 269: 413-426
        • Golder S.
        • Loke Y.K.
        • Zorzela L.
        Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews.
        Health Inf libraries J. 2014; 31: 92-105