Advertisement

Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study

  • Hannah Ewald
    Correspondence
    Corresponding author. University Medical Library, University of Basel, Spiegelgasse 5, CH-4051 Basel, Switzerland. Tel.: +41 (0)61 2075564.
    Affiliations
    University Medical Library, University of Basel, Basel, Switzerland

    Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
    Search for articles by this author
  • Irma Klerings
    Affiliations
    Cochrane Austria, Danube University Krems, Krems a.d. Donau, Austria
    Search for articles by this author
  • Gernot Wagner
    Affiliations
    Cochrane Austria, Danube University Krems, Krems a.d. Donau, Austria
    Search for articles by this author
  • Thomas L. Heise
    Affiliations
    Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany

    Institute for Public Health and Nursing Research - IPP, Health Sciences Bremen, University of Bremen, Bremen, Germany
    Search for articles by this author
  • Jan M. Stratil
    Affiliations
    Institute for Medical Informatics, Biometry and Epidemiology, Pettenkofer School of Public Health, LMU Munich, Munich, Germany
    Search for articles by this author
  • Stefan K. Lhachimi
    Affiliations
    Institute for Public Health and Nursing Research - IPP, Health Sciences Bremen, University of Bremen, Bremen, Germany

    Department of Health, Nursing, Management, University of Applied Sciences Neubrandenburg, 17033, Neubrandenburg, Germany
    Search for articles by this author
  • Lars G. Hemkens
    Affiliations
    Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland

    Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA

    Meta-Research Innovation Center Berlin (METRIC-B), Berlin Institute of Health, Berlin, Germany
    Search for articles by this author
  • Gerald Gartlehner
    Affiliations
    Cochrane Austria, Danube University Krems, Krems a.d. Donau, Austria

    RTI International, Research Triangle Park, NC, USA
    Search for articles by this author
  • Susan Armijo-Olivo
    Affiliations
    University of Applied Sciences Osnabrück, Faculty of Economics and Social Sciences – Osnabrück, Germany

    Faculties of Rehabilitation Medicine and Medicine and Dentistry, University of Alberta, Edmonton, Canada
    Search for articles by this author
  • Barbara Nussbaumer-Streit
    Affiliations
    Cochrane Austria, Danube University Krems, Krems a.d. Donau, Austria
    Search for articles by this author
Open AccessPublished:May 30, 2022DOI:https://doi.org/10.1016/j.jclinepi.2022.05.022

      Abstract

      Background and Objectives

      Assessing changes in coverage, recall, review, conclusions and references not found when searching fewer databases.

      Methods

      In randomly selected 60 Cochrane reviews, we checked included study publications' coverage (indexation) and recall (findability) using different search approaches with MEDLINE, Embase, and CENTRAL and related them to authors' conclusions and certainty. We assessed characteristics of unfound references.

      Results

      Overall 1989/2080 included references, were indexed in ≥1 database (coverage = 96%). In reviews where using one of our search approaches would not change conclusions and certainty (n = 44-54), median coverage and recall were highest (range 87.9%-100.0% and 78.2%-93.3%, respectively). Here, searching ≥2 databases reached >95% coverage and ≥87.9% recall. In reviews with unchanged conclusions but less certainty (n = 2-8): 63.3%-79.3% coverage and 45.0%-75.0% recall. In reviews with opposite conclusions (n = 1-3): 63.3%-96.6% and 52.1%-78.7%. In reviews where a conclusion was no longer possible (n = 3-7): 60.6%-86.0% and 20.0%-53.8%. The 265 references that were indexed but unfound were more often abstractless (30% vs. 11%) and older (28% vs. 17% published before 1991) than found references.

      Conclusion

      Searching ≥2 databases improves coverage and recall and decreases the risk of missing eligible studies. If researchers suspect that relevant articles are difficult to find, supplementary search methods should be used.

      Keywords

      What is new?

        Key findings

      • Searching at least two databases improves coverage and recall and decreases the chances of making an inappropriate conclusion in a review.

        What this adds to what is known

      • A systematic review not always needs to find all eligible studies to arrive at the same conclusion with the same certainty as compared to including all eligible studies. However, in some cases even combined searches of MEDLINE, Embase, and CENTRAL to identify research may not be sufficient to draw any conclusion.
      • Many studies, almost a third, which were indexed but not found by searching MEDLINE, Embase and CENTRAL had no abstract.

        What is the implication, what should change now?

      • In most cases, searching in a limited number of databases will lead to the same conclusions in comprehensive systematic reviews of RCTs.
      • If researchers suspect that some relevant articles may be difficult to find or have no abstract, supplementary strategies rather than additional databases should be used.
      • Researchers should carefully pick suitable electronic databases especially with respect to topic and relevant subject headings or consult an information specialist.

      1. Introduction

      Systematic reviews are crucial to inform healthcare decision-making. Their purpose is to identify, appraise, and synthesize all evidence on a specific research question using explicit, systematic methods [
      • Higgins J.P.T.
      • Thomas J.
      • Chandler J.
      • Cumpston M.
      • Li T.
      • Page M.J.
      • et al.
      Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021).
      ]. A comprehensive systematic literature search is the basis for every systematic review. It aims to reduce bias in study selection by using a variety of information sources. These searches usually comprise of searching multiple bibliographic databases and supplementary search methods (hand searches, searches in study registers, web searching, etc.). The Methodological Expectations of Cochrane Intervention Reviews (MECIR) require searching at least MEDLINE, CENTRAL, and Embase (depending on access availability), study registers as well as reference lists of included studies and relevant systematic reviews [
      • Higgins J.P.T.
      • Lasserson T.
      • Chandler J.
      • Tovey D.
      • Thomas J.
      • Flemyng E.
      • et al.
      Methodological expectations of Cochrane intervention reviews.
      ].
      Recent methods studies evaluating research studies of medical or psychological topics [
      • Hartling L.
      • Featherstone R.
      • Nuspl M.
      • Shave K.
      • Dryden D.M.
      • Vandermeer B.
      The contribution of databases to the results of systematic reviews: a cross-sectional study.
      ,
      • Halladay C.W.
      • Trikalinos T.A.
      • Schmid I.T.
      • Schmid C.H.
      • Dahabreh I.J.
      Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions.
      ,
      • Frandsen T.F.
      • Eriksen M.B.
      • Hammer D.M.G.
      • Christensen J.B.
      PubMed coverage varied across specialties and over time: a large-scale study of included studies in Cochrane reviews.
      ] found that the majority of relevant references included in Cochrane reviews were available (indexed) in MEDLINE (i.e., database coverage). Removing studies that were not indexed in MEDLINE from the meta-analyses had only little impact on the effect estimates [
      • Hartling L.
      • Featherstone R.
      • Nuspl M.
      • Shave K.
      • Dryden D.M.
      • Vandermeer B.
      The contribution of databases to the results of systematic reviews: a cross-sectional study.
      ,
      • Halladay C.W.
      • Trikalinos T.A.
      • Schmid I.T.
      • Schmid C.H.
      • Dahabreh I.J.
      Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions.
      ,
      • Marshall I.J.
      • Marshall R.
      • Wallace B.C.
      • Brassey J.
      • Thomas J.
      Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study.
      ]. However, these results may be overestimating the impact of searching only one database as they have not assessed the recall of the indexed studies. Recall is, in this context, a more meaningful measure: It indicates the amount of relevant indexed references that were actually found by the search strategy in a particular database. Comprehensive searches for systematic reviews aim for high recall by applying broad searches to find all relevant references. As this usually retrieves many irrelevant references, practical concerns require balancing the sensitivity of the searches with the available time and budget [
      • Higgins J.P.T.
      • Thomas J.
      • Chandler J.
      • Cumpston M.
      • Li T.
      • Page M.J.
      • et al.
      Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021).
      ].
      Combining multiple information sources often improves recall compared to single-database searches [
      • Aagaard T.
      • Lund H.
      • Juhl C.
      Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders?.
      ,
      • Bramer W.M.
      • Giustini D.
      • Kramer B.M.
      Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study.
      ,
      • Bramer W.M.
      • Rethlefsen M.L.
      • Kleijnen J.
      • Franco O.H.
      Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study.
      ,
      • Levay P.
      • Raynor M.
      • Tuvey D.
      The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE public health guidance.
      ]. Bramer et al. [
      • Bramer W.M.
      • Giustini D.
      • Kramer B.M.
      Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study.
      ] assessed systematic search results in various medical topics, from therapeutic effectiveness and diagnostic accuracy to ethics and public health. They found high database coverage in MEDLINE and Embase, but reported that in some cases even the cumulative recall of MEDLINE and Embase was below 50%.
      This project is part of a larger methodological project that examined how conclusions of 60 published Cochrane systematic reviews [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ,
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Titscher V.
      • Gartlehner G.
      Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study.
      ] and treatment effect estimates [
      • Ewald H.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated and comprehensive literature searches led to identical or very similar effect estimates: a meta-epidemiological study.
      ] changed when the number of information sources is limited (details below). Here, we extend the published results by focusing on database coverage and recall and possible reasons as to why some studies cannot be retrieved anymore.
      Our aim was to assess how coverage and recall relate to conclusions in reviews and why some references could not be found anymore when searching in a limited number of databases.

      2. Methods

      2.1 Methods of the main study

      Here we briefly summarize the most important methodological aspects of the underlying project previously reported in detail [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ,
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Titscher V.
      • Gartlehner G.
      Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study.
      ,
      • Ewald H.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated and comprehensive literature searches led to identical or very similar effect estimates: a meta-epidemiological study.
      ]. We randomly selected 60 Cochrane reviews that included a replicable search strategy, had clear conclusions, and reported a meta-analysis [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Titscher V.
      • Gartlehner G.
      Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study.
      ]. We determined the clinical topic of each review and categorized all reviews according to the type of intervention (pharmacological or nonpharmacological). From each review, we used all references reported in the “References to studies included in this review” section as a gold standard-set to assess 14 different search approaches using a limited number of sources (i.e., the 14 possible combinations of MEDLINE, Embase, CENTRAL with and without screening reference lists). We identified which studies were not found when we limited the number of databases and recalculated the estimates from meta-analyses for all outcomes of the main summary of findings table for each review accordingly [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ,
      • Ewald H.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated and comprehensive literature searches led to identical or very similar effect estimates: a meta-epidemiological study.
      ]. Then, we asked the original authors whether the resulting summary of findings tables would alter their conclusions. Possible answers were (1) conclusion does not change (“same conclusion/same certainty”), (2) same conclusion but with less certainty, (3) conclusion with opposite direction (“opposite conclusion”), and (4) conclusion not possible anymore [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ].

      3. Methods of this sub-study

      For this sub-study, we used the same gold standard-set as mentioned above. We first determined what the individual and cumulative database coverage and recall would have been, had the searches been done in a limited number of databases or database combinations, namely: (1) MEDLINE-only (M), (2) Embase-only (E), (3) CENTRAL-only (C); (4) MEDLINE + Embase (M + E), (5) MEDLINE + CENTRAL (M + C), (6) Embase + CENTRAL (E + C), and (7) MEDLINE + Embase + CENTRAL (M + E + C).
      We then put database coverage and recall of these limited searches in relation to changes in review conclusions. Finally, we assessed indexation status, publication date, abstract availability, language, intervention type and topic of references that could not be found.
      As the focus of this study is coverage and recall, we did not consider search combinations with reference list checking.
      From the Cochrane reviews and our previous study [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ], we extracted year of publication, publication type, study design, topic, and intervention types (pharmacological or nonpharmacological) for each reference included in the gold standard-set. From studies not found, we specifically checked indexation status, abstract availability, publication date, language, and topic.

      3.1 Data analysis

      We used the Endnote X8 (Clarivate Analytics) and Excel 2016 (Microsoft) for data management and analysis. For each review, we saved the included studies to EndNote and exported the bibliographic data as tables to Excel. For each reference, we noted the availability in the target databases using the database-specific identifiers. We used these identifiers to determine if available studies were found by the original search strategies.
      We calculated database coverage and recall as ratios (Appendix 1): The numerator for database coverage is the number of relevant references indexed in a database, and the numerator for recall is the individual or cumulative number of relevant references found in a database or a database combination, respectively, at the time of the original search. As denominator for both these metrics, we used the gold standard-set.
      To evaluate overall coverage and recall, we performed descriptive statistical analyses for each of the seven search approaches. We calculated the mean and the median and interquartile range (IQR) across all reviews. We contrasted coverage and recall of the search approaches to the conclusions of the respective Cochrane reviews (“same conclusion/same certainty”, “same conclusion/less certainty”, “opposite conclusion”, “no conclusion possible”). We also analyzed these results by intervention types, i.e., pharmaceutical and nonpharmaceutical.
      We examined references that were not found in at least one of the three databases with respect to their indexation status, publication date, abstract availability, language, intervention type, and topic. Between indexed references that were found with one of the limited database search approaches and those that were not found, we compared abstract availability and publication date (before 1991/1991 or later; rationale for date cut-off: Introduction of “Randomized Controlled Trial” and “Clinical Trial” as Publication Types in Medline in 1991).

      4. Results

      4.1 Study characteristics

      Each of the 60 Cochrane reviews used between 2 and 18 (median 6) information sources in addition to M + E + C (including other databases, grey literature, citation tracking [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ]). Fifty-six reviews included solely randomized controlled trials (RCTs) or quasi-RCTs; four reviews additionally included nonrandomized controlled trials or observational studies (designs as defined by Cochrane review authors). The review characteristics (number of reviews, included studies, and proportion of pharmacological interventions) are reported in relation to changes in reviews' conclusions in Appendix 2.
      The 60 Cochrane reviews included 2,080 references of studies (the gold standard-set) published between 1960 and 2016: 2014 (96.8%) were references to RCTs or quasi-randomized trials. Most references were journal articles (1,807/2,080, 87%), although 273 (13.1%) referred to other publication types (e.g., conference abstracts, study registries; Table 1).
      Table 1Characteristics of references of studies included by the 60 Cochrane reviews
      CharacteristicsSubcategoryReferences
      2,080 (100%)
      InterventionPharmacological1,052 (50.6%)
      Nonpharmacological1,028 (49.4%)
      Study designRCT/quasi-RCT2,014 (96.8%)
      Other design
      non-randomized controlled clinical trial, controlled cohort study, before and after study, interrupted time series.
      66 (3.2%)
      Publication date1960 to 1990375 (18.0%)
      1991 to 20161,671 (80.3%)
      Undated document34 (1.6%)
      Publication typeJournal article1,807 (86.9%)
      Conference abstract180 (8.7%)
      Study register entry62 (3.0%)
      Pharma
      Documents provided by pharmaceutical companies.
      4 (0.2%)
      Book3 (0.1%)
      Thesis19 (0.9%)
      Other
      Other unpublished documents.
      5 (0.2%)
      Abbreviation: RCT, randomized controlled trial.
      a non-randomized controlled clinical trial, controlled cohort study, before and after study, interrupted time series.
      b Documents provided by pharmaceutical companies.
      c Other unpublished documents.

      4.2 Coverage

      Of the 2,080 references in the gold standard-set, 1,989 (95.6%) were indexed in at least one of the three databases (median coverage = 98.5%; Table 2).
      Table 2Database coverage and recall of all references included in the 60 Cochrane reviews and stratified according to pharmacological and non-pharmacological interventions
      Database combinationCoverage N ReferencesDatabase coverage in %Recall N ReferencesRecall in %
      MeanMedian (IQR) per reviewMeanMedian (IQR) per review
      Gold standard-set2,0802,080
      MEDLINE1,65079.387.0(72.2-93.0)1,35465.175.5(56.8-87.0)
      Embase1,60277.083.0(62.6-92.4)1,27661.371.0(46.5-84.3)
      CENTRAL1,83988.491.3(75.0-100.0)1,50272.275.0(65.1-92.6)
      M + E1,80987.094.1(75.0-100.0)1,53173.685.4(66.6-95.1)
      M + C1,94293.495.7(88.5-100.0)1,62878.385.5(74.0-94.2)
      C + E1,95994.296.6(88.7-100.0)1,65679.689.0(72.6-98.4)
      M + E + C1,98995.698.5(91.9-100.0)1,72482.990.0(77.7-100.0)
      Pharmacological interventions (n = 30)
      Gold standard-set1,0521,052
      MEDLINE79976.085.7(68.8-89.6)74570.875.7(62.8-86.6)
      Embase84280.086.6(66.3-94.4)76772.974.6(62.5-83.7)
      CENTRAL96791.994.0(86.2-100.0)87082.782.3(72.2-95.8)
      M + E89284.890.5(76.8-97.3)85080.884.5(71.3-95.4)
      M + C99494.596.7(89.4-100.0)90786.288.4(75.7-95.6)
      C + E1,01696.696.9(93.7-100.0)96091.393.3(82.4-100.0)
      M + E + C1,01896.898.8(93.7-100.0)96691.894.1(82.4-100.0)
      Nonpharmacological interventions (n = 30)
      Gold standard-set1,0281,028
      MEDLINE85182.889.6(73.5-95.6)60959.275.5(52.2-88.4)
      Embase76073.982.0(60.7-91.2)50849.461.3(30.5-83.3)
      CENTRAL87284.881.2(70.1-100.0)63261.570.7(55.7-89.6)
      M + E91789.295.9(78.8-100.0)68066.185.1(60.1-94.4)
      M + C94892.294.7(86.4-100.0)72170.180.5(64.0-92.9)
      C + E94391.796.3(81.8-100.0)69567.680.6(59.4-93.9)
      M + E + C97194.598.5(91.8-100.0)75973.787.0(64.0-97.7)
      Abbreviations: C, CENTRAL; E, Embase; IQR, Interquartile range; M, MEDLINE; Gold standard-set, references included in the 60 Cochrane reviews using the original search methods; Mean, The total number of included references indexed or found in a database divided by the total number of included references found by the comprehensive Cochrane review search; Median, The median value of database coverage or recall per review; N, number of.
      When limiting the number of databases, single-database coverage across all 60 Cochrane reviews was highest in CENTRAL (median: 91.3%, IQR: 75.0%-100.0%). The median coverage in CENTRAL and EMBASE was higher for reviews on pharmacological than nonpharmacological interventions although this was the opposite for MEDLINE (Table 2).
      The combination of databases increased the median coverage. The highest coverage could be attained with M + E + C (Median 98.5%; IQR 91.9%-100.0%) (Table 2).
      For nearly half of the reviews under investigation (n = 29; 16 pharmacological, 13 nonpharmacological), there was at least one database or combination with a coverage of 100% (Appendix 3).

      4.3 Recall

      Of the 2,080 references in the gold standard-set, 1,724 (83%) could be found searching in M + E + C (Table 2).
      Across all 60 Cochrane reviews, recall was markedly lower than coverage. The highest median single-database recall was in MEDLINE (median: 75.5%, IQR: 56.8%-87.0%, Table 2) which seemed balanced between reviews of pharmacological and nonpharmacological interventions. CENTRAL and Embase median recall was higher for reviews on pharmacological interventions (Table 2).
      There was a gradual increase in recall with more databases included across evaluated search approaches. The highest median recall was 90.0% in M + E + C (IQR: 77.7%-100.0%; pharmacological interventions: n = 30, median 94.1%, IQR 82.4%-100.0%; nonpharmacological interventions: n = 30, median 87.0%, IQR 64.0%-97.7%) (Table 2).
      In 18 Cochrane reviews (11 pharmacological, 7 nonpharmacological), at least one search strategy or combination had a recall of 100% (Appendix 3).

      4.4 Coverage in relation to reviews' conclusions

      Table 3 shows how database coverage relates to the conclusions of the Cochrane review when searching in a limited number of databases.
      Table 3Database coverage in relation to search approach and conclusion category
      ConclusionsMEDLINE (M)Embase (E)CENTRAL (C)
      N reviewsReferencesCoverage per review in %N reviewsReferencesCoverage per review in %N reviewsReferencesCoverage per review in %
      Incl.Index.MeanMedian (IQR)Incl.Index.MeanMedian (IQR)Incl.Index.MeanMedian (IQR)
      Same conclusion, same certainty481,7531,39779.787.9 (76.1-93.7)441,5611,24479.787.9 (74.6-93.5)471,7621,61191.496.2 (85.2-100.0)
      Same conclusion, less certainty6875866.763.3 (57.3-78.6)622015972.366.0 (52.5-76.1)819913366.867.8 (62.7-78.5)
      Opposite conclusion21199176.563.3 (50.0-76.6)314511075.979.2 (55.2-85.1)1898191.091.0 (N/A)
      No conclusion possible412110486.084.0 (65.9-94.7)71548857.162.5 (48.8-81.3)4301446.760.6 (37.7-81.3)
      M + EM + CC + E
      Same conclusion, same certainty501,7941,56287.195.3 (86.3-100.0)531,9121,80294.296.4 (90.0-100.0)501,8161,73995.898.0 (92.0-100.0)
      Same conclusion, less certainty5765369.771.9 (57.1-73.7)3574273.771.9 (68.1-81.4)615312179.179.3 (71.2-88.2)
      Opposite conclusion1898696.696.6 (N/A)1898494.494.4 (N/A)1898696.696.6 (N/A)
      No conclusion possible412110889.386.0 (65.9-97.7)3221463.675.0 (64.4-87.5)3221359.175.0 (60.6-87.5)
      M + E + C
      Same conclusion, same certainty541,9231,85396.4100.0 (93.5-100.0)
      Same conclusion, less certainty2463678.378.3 (78.2-78.5)
      Opposite conclusion1898696.696.6 (N/A)
      No conclusion possible3221463.675.0 (64.4-87.5)
      Abbreviations: C, CENTRAL; E, Embase; Incl., References included by reviews; Index., Reverences indexed in databases; IQR, Interquartile range; M, MEDLINE; Mean, The total number of included references indexed in a database divided by the total number of included references; Median: The median value of database coverage per review; N/A, not applicable (only 1 review).
      In reviews where the conclusion and certainty did not change (depending on search approach n = 44 to 54 of 60), CENTRAL and each combination of at least two databases reached over 95% median coverage (although a mean coverage of at least 95% was only reached with C + E and M + E + C).
      In reviews where the conclusion did not change but the authors were less certain (n = 2 to 8 of 60), median coverage was much lower ranging from 63.3% to 78.3% from single database searches to M + E + C (Table 3).
      When limiting the information sources to M + E + C, the conclusion did not change (i.e., with the same or less certainty) in 56 reviews including 1,969 references. Half of these reviews (28/56; 50%) included at least one reference that was not indexed and hence not found (80/1,969, 4%; coverage = 78.2%-78.5%; Table 3).
      In reviews where searching in a limited number of databases led to an opposite conclusion (n = 1 to 3 of 60), median coverage ranged between 63.3% (M) and 96.6% (C + E, M + E, M + E + C) (Table 3).
      In reviews where a conclusion could no longer be drawn (n = 3 to 7 of 60 reviews), we found the lowest median coverage which ranged from 60.6% (C) to 86.0% (M + E).
      When limiting the information sources to M + E + C, a conclusion was no longer possible or changed to the opposite of the original review in three [
      • Geretsegger M.
      • Elefant C.
      • Mössler Karin A.
      • Gold C.
      Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews.
      ,
      • McNamara D.A.
      • Goldberger J.J.
      • Berendsen M.A.
      • Huffman M.D.
      Implantable defibrillators versus medical therapy for cardiac channelopathies.
      ,
      • Wong G.W.
      • Wright J.M.
      Blood pressure lowering efficacy of nonselective beta-blockers for primary hypertension.
      ] and one [
      • Blessberger H.
      • Kammler J.
      • Domanovits H.
      • Schlager O.
      • Wildner B.
      • Azar D.
      • et al.
      Perioperative beta-blockers for preventing surgery-related mortality and morbidity.
      ] reviews, respectively. Median coverage of these reviews ranged between 64.4% and 96.6%. Three of these four reviews included at least one reference that was not indexed (overall 11 of 111, 10%), but coverage differed greatly between individual reviews (54%-100%; Appendix 5).

      4.5 Recall in relation to reviews' conclusions

      Analysis of recall by conclusion (Table 4) showed that search approaches that led to the same conclusion with the same certainty had a markedly higher median recall than those where any change occurred (“same direction but less certainty”, “opposite conclusion” or “no conclusion possible”). The category “no conclusion possible” had the lowest median recall across all limited database search approaches (median ranged from 20.0% to 53.8%). Depending on which approach was used (single database search, two databases, or combination of all three), there were always three to seven Cochrane reviews where a conclusion was no longer possible (median recall ranged from 20.0% to 53.8%) and one to three reviews where the opposite conclusion was drawn (median recall ranged from 52.1% to 78.7%).
      Table 4Recall in relation to search approach and conclusion category
      ConclusionsMEDLINE (M)Embase (E)CENTRAL (C)
      N reviewsN ReferencesRecall per review in %N reviewsN ReferencesRecall per review in %N reviewsN ReferencesRecall per review in %
      Incl.FoundMeanMedian (IQR)Incl.FoundMeanMedian (IQR)Incl.FoundMeanMedian (IQR)
      Same conclusion, same certainty481,7531,20868.981.4 (66.7-88.1)441,5611,11271.278.2 (62.5-88.1)471,7621,31374.583.3 (70.2-95.7)
      Same conclusion, less certainty6875158.650.0 (48.0-64.1)62204922.345.0 (31.4-57.5)819911155.859.5 (47.6-66.7)
      Opposite conclusion21197361.352.1 (42.7-61.4)31458558.666.3 (44.1-72.7)1896573.073.0 (N/A)
      No conclusion possible41212218.221.4 (9.1-41.8)71542918.820.0 (13.8-35.4)4301343.329.3 (9.4-53.4)
      M + EM + CC + E
      Same conclusion, same certainty501,7941,38777.388.6 (73.8-97.1)531,9121,50878.987.9 (75.0-95.1)501,8161,48481.792.5 (79.2-100.0)
      Same conclusion, less certainty5764660.550.0 (50.0-57.9)3574171.975.0 (66.1-78.4)61539058.860.4 (50.5-67.3)
      Opposite conclusion1897078.778.7 (N/A)1896674.274.2 (N/A)1897078.778.7 (N/A)
      No conclusion possible41212823.124.5 (13.6-41.8)3221359.153.8 (26.9-64.4)3221254.546.2 (23.1-60.6)
      M + E + C
      Same conclusion, same certainty541,9231,60883.693.3 (85.0-100.0)
      Same conclusion, less certainty2463371.767.6 (62.4-72.9)
      Opposite conclusion1897078.778.7 (N/A)
      No conclusion possible3221359.153.8 (26.9-64.4)
      Abbreviations: Incl., References included by reviews; IQR, interquartile range; N/A, not applicable (only 1 review).
      When limiting the information sources to M + E + C, the conclusion did not change (i.e., with the same or less certainty) in 56 reviews including 1,969 references. A sixth of these references had not been found (328/1,969, 17%). Median recall per review ranged between 62.4% and 100% (Table 4). Of the 1,969 references, 1,706 were journal articles and 263 grey literature articles of which 238 (14%) and 90 (34%) were not found, respectively. The 90/263 grey literature references that could not be found when limiting the information sources to M + E + C and the conclusion remained unchanged were 15/60 study register entries (25%), 52/177 conference abstracts (29%), 11/14 theses (79%), 3/3 books (100%), 4/4 pharma documents (100%), and 5/5 other document types (100%). Of the 111 study references (101 journal articles and 10 grey literature articles) included in the four reviews where a conclusion was no longer possible or changed to the opposite of the original review, 28 were not found (Appendix 4).

      4.6 Characteristics of references not found

      Overall, 356 of the 2,080 (17%) references were not found when using any of the seven search approaches in a limited number of databases.
      To specifically assess the nature of these references, we removed 30 exact duplicates, i.e., references that were included in two different Cochrane reviews [
      • Hooper L.
      • Martin N.
      • Abdelhamid A.
      • Davey Smith G.
      Reduction in saturated fat intake for cardiovascular disease.
      ,
      • Hooper L.
      • Abdelhamid A.
      • Bunn D.
      • Brown T.
      • Summerbell C.D.
      • Skeaff C.M.
      Effects of total fat intake on body weight.
      ] or twice in the same Cochrane review [
      • Hooper L.
      • Martin N.
      • Abdelhamid A.
      • Davey Smith G.
      Reduction in saturated fat intake for cardiovascular disease.
      ,
      • Birks Jacqueline S.
      • Chong Lee Y.
      • Grimley Evans J.
      Rivastigmine for Alzheimer’s disease.
      ]. Of the 326 individual references not found, 91 (28%) were not indexed in MEDLINE, Embase or CENTRAL. These were mainly grey literature (36/91 conference abstracts, 14/91 study registers, 12/91 theses, and 9/91 other types) and some journal articles (20/91, 22%). Of the remaining 235 individual, not found, indexed references, the majority had an abstract (70%), and was written in English (99%). The medical topics were mainly cardiovascular diseases (74%) and the intervention types were nutrition or physical activity (62%) (Table 5).
      Table 5Characteristics of the 326 individual study references not found when limiting database searches to MEDLINE, Embase and/or CENTRAL
      CharacteristicsOf the 326:Not indexedIndexed<1991No abstract in databaseNot EnglishPsychological/EducationalNutrition/Physical activityPharma-cologicalComplexOtherCerebro-vascular:Chronic respi-ratoryCVDMental healthOsteo-arthritis
      Not indexed91 (28%)N/A7913221134141012237265
      IndexedN/A235 (72%)737124145511421230174236
      <19918%31%80 (25%)390143161190205451
      No abstract in database100%30%49%162 (50%)4225350231422892346
      Not English3%1%0%2%5 (2%)1120101211
      Intervention typePsychological/Educational2%2%1%14%20%26 (8%)N/AN/AN/AN/A007181
      Nutrition/Physical activity12%62%54%33%20%N/A156 (48%)N/AN/AN/A0814062
      Pharmacological37%22%20%31%40%N/AN/A85 (26%)N/AN/A31650124
      Complex15%6%1%14%0%N/AN/AN/A28 (9%)N/A0213130
      Other11%9%24%9%20%N/AN/AN/AN/A31 (10%)026104
      TopicCerebrovascular:1%1%0%1%0%0%0%4%0%0%3 (1%)N/AN/AN/AN/A
      Chronic respiratory24%13%25%17%20%0%5%19%7%84%N/A52 (16%)N/AN/AN/A
      CVD41%74%68%57%40%27%90%59%46%0%N/AN/A211 (65%)N/AN/A
      Mental Health29%10%6%21%20%69%4%14%46%0%N/AN/AN/A49 (15%)N/A
      Osteoarthritis6%3%1%4%20%4%1%5%0%13%N/AN/AN/AN/A11 (3%)
      Italics: points where row and column are identical, highlighted for readability. Absolute numbers of studies are shown in the upper right section, correlating percentages are shown in the lower left section.
      In comparison to the references that could be found with at least one of the limited database search approaches, the references that were not found were on average older (28% vs. 17% published before 1991) and more often had no abstract available (30% vs. 11%) (Appendix 5).

      5. Discussion

      5.1 Overall results

      The majority of references included in a random sample of 60 Cochrane reviews were indexed in at least one of the databases MEDLINE, Embase, and CENTRAL. We found a median coverage of 94%-99% for every combination of at least two databases. As expected, searching in more databases improved recall, but not all indexed references could be found. The median recall was 85%-90% for every combination of at least two databases. In comparison, median coverage of a single database ranged between 83% and 91%, and median recall between 71% and 76%. The overall database coverage and recall observed in our study are similar to findings of previous research [
      • Aagaard T.
      • Lund H.
      • Juhl C.
      Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders?.
      ,
      • Bramer W.M.
      • Giustini D.
      • Kramer B.M.
      Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study.
      ,
      • Bramer W.M.
      • Rethlefsen M.L.
      • Kleijnen J.
      • Franco O.H.
      Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study.
      ].
      We could not conclusively answer the question on how many databases should be searched. Although searching two databases may be enough for some systematic reviews, others may require a more thorough approach like MECIR. However, we identified some factors that seem to play a role: When we stratified according to pharmacological and nonpharmacological interventions, we found that although median coverage of studies with pharmacological interventions was slightly higher than with nonpharmacological interventions in Embase, and CENTRAL, all databases had a higher mean and median recall of studies with pharmacological interventions than of studies with nonpharmacological interventions. This could indicate that successfully finding studies may depend on the intervention-type or topic: pharmacological interventions can be well-described and delimited by using the drugs’ generic names, trade names, and codes. Nonpharmacological interventions are often more complex (e.g., psychological interventions) and the multiplicity of possible terms may be more difficult to capture in a sensitivity-specificity balanced search strategy. Depending on the topic, this could be especially aggravated by missing granularity or even gaps in subject headings [
      • Minguet F.
      • Van Den Boogerd L.
      • Salgado T.M.
      • Correr C.J.
      • Fernandez-Llimos F.
      Characterization of the medical subject headings thesaurus for pharmacy.
      ], which gives reason to carefully pick suitable electronic databases and subject headings [
      • Barroso J.
      • Gollop C.J.
      • Sandelowski M.
      • Meynell J.
      • Pearce P.F.
      • Collins L.J.
      The challenges of searching for and retrieving qualitative studies.
      ] and/or consult an information specialist [
      • Higgins J.P.T.
      • Thomas J.
      • Chandler J.
      • Cumpston M.
      • Li T.
      • Page M.J.
      • et al.
      Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021).
      ,
      • Kugley S.
      • Wade A.
      • Thomas J.
      • Mahood Q.
      • Jørgensen A.-M.K.
      • Hammerstrøm K.
      • et al.
      Searching for studies: a guide to information retrieval for Campbell systematic reviews.
      ,
      • Rethlefsen M.L.
      • Murad M.H.
      • Livingston E.H.
      Engaging medical librarians to improve the quality of review articles.
      ].
      We found that references of Cochrane reviews where the conclusion was no longer possible tended to have lower median database coverage than the others, although Cochrane reviews where the conclusion and its certainty remained the same had a markedly higher median coverage than those where changes occurred. However, on an individual level, there is no consistent pattern of low coverage and conclusion (Appendix 4). In two instances [
      • Geretsegger M.
      • Elefant C.
      • Mössler Karin A.
      • Gold C.
      Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews.
      ,
      • Wong G.W.
      • Laugerotte A.
      • Wright J.M.
      Blood pressure lowering efficacy of dual alpha and beta blockers for primary hypertension.
      ], all references that were indexed were also found (coverage = recall), but the conclusion was still not possible. In such cases it would have been necessary to find the nonindexed references. As our previous study has shown, combining the database search with reference list checking would already have helped to accurately determine conclusions [
      • Nussbaumer-Streit B.
      • Klerings I.
      • Wagner G.
      • Heise T.L.
      • Dobrescu A.I.
      • Armijo-Olivo S.
      • et al.
      Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
      ]. Hence, researchers should consider using supplementary search techniques that are less dependent on indexation, such as citation tracking or contacting experts in the field [
      • Aagaard T.
      • Lund H.
      • Juhl C.
      Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders?.
      ,
      • Cooper C.
      • Booth A.
      • Britten N.
      • Garside R.
      A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
      ,
      • Vassar M.
      • Yerokhin V.
      • Sinnett P.M.
      • Weiher M.
      • Muckelrath H.
      • Carr B.
      • et al.
      Database selection in systematic reviews: an insight through clinical neurology.
      ].
      Almost a fifth of all references were not found when using any of the seven search approaches in a limited number of databases. It may not always be necessary to find all the relevant references do draw a “correct” conclusion. However, we do not know when the necessary studies are found and measures should be taken to optimize the search and tailor the search approach to the research topic. A third of the references in our study that were indexed in at least one database but were not found had no abstract. As most databases only allow searching in the metadata of a study (especially in title, abstract, author keywords and subject headings) rather than the full text, a reference without an abstract is harder to find and more dependent on the use of subject headings such as MeSH and Emtree. To find such articles, one would have to improve the performance of searches. This includes database choice and supplementary search techniques. Another aspect is the quality and adequacy of the searches. Amongst others, these can be improved by using text analysis to inform search strategies [
      • Hausner E.
      • Guddat C.
      • Hermanns T.
      • Lampert U.
      • Waffenschmidt S.
      Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one.
      ], using systematic approaches to build comprehensive searches [
      • Bramer W.M.
      • de Jonge G.B.
      • Rethlefsen M.L.
      • Mast F.
      • Kleijnen J.
      A systematic approach to searching: an efficient and complete method to develop literature searches.
      ,
      • Hausner E.
      • Guddat C.
      • Hermanns T.
      • Lampert U.
      • Waffenschmidt S.
      Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
      ], and peer review of search strategies [
      • McGowan J.
      • Sampson M.
      • Salzwedel D.M.
      • Cogo E.
      • Foerster V.
      • Lefebvre C.
      PRESS peer review of electronic search strategies: 2015 guideline statement.
      ].

      5.2 Limitations

      This study has some limitations. First, the relationship between recall and conclusions is not straightforward. As Cochrane reviews list all included study publications, it is possible that references not found by the searches belonged to studies included in the narrative syntheses, not the meta-analyses. In that case, they affected the recall of the searches but not the conclusions of the review. A future analysis could stratify results by a Cochrane review's “primary included study”. In addition, we calculated the recall independently of the database coverage. This means that even a search strategy that found all indexed references would have a low recall if the review contained many nonindexed references.
      Second, when determining if records were indexed in the databases, we limited results to MEDLINE-only and Embase-only, to ensure the results were independent of the search interface. However, this might differ from the actual search experience as some interfaces index additional documents, e.g., PubMed includes MEDLINE-in-Process materials and nonMEDLINE records, and Embase.com includes most MeSH-indexed MEDLINE records. Hence, we may have effectively underestimated the coverage and recall of searching in a limited number of databases. Conversely, by using references from Cochrane reviews, we might overestimate the database coverage of CENTRAL because references to RCTs included in Cochrane reviews are regularly added to this database. We sought to counter this issue by verifying that the references were available before the date of the latest search. However, this does not account for updated reviews, where some references might have been added after the publication of the original review and then picked up by the update searches.
      Third, most Cochrane reviews' search strategies are designed by trained information specialists who follow the Cochrane handbook [
      • Higgins J.P.T.
      • Thomas J.
      • Chandler J.
      • Cumpston M.
      • Li T.
      • Page M.J.
      • et al.
      Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021).
      ] and MECIR standards [
      • Higgins J.P.T.
      • Lasserson T.
      • Chandler J.
      • Tovey D.
      • Thomas J.
      • Flemyng E.
      • et al.
      Methodological expectations of Cochrane intervention reviews.
      ]. We assumed that these would be comprehensive and of high quality to serve as gold-standard [
      • Hausner E.
      • Guddat C.
      • Hermanns T.
      • Lampert U.
      • Waffenschmidt S.
      Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
      ]. Hence, we only checked the search strategies for formal errors (i.e., syntax, spelling, block building), not for comprehensiveness of search terms or overall quality. Although a low recall could relate to suboptimal searches [
      • Franco J.V.A.
      • Garrote V.L.
      • Escobar Liquitay C.M.
      • Vietto V.
      Identification of problems in search strategies in Cochrane Reviews.
      ], it is also possible that a good quality search was designed to be specific and hence, a lower recall would have been a deliberate choice.

      6. Conclusion

      Database coverage alone cannot predict whether relevant references will be found (=recall) by a search strategy. Cumulative search results of two or more databases improve coverage and recall and decrease the chances of making an inappropriate conclusion in a review. Yet, even the combination of the three most frequently used databases Medline, Embase, and CENTRAL did not suffice to achieve total recall or avoid any change in the conclusions of systematic reviews. For some research topics a small number of database searches combined with supplementary search methods may be more useful than searching many bibliographic databases. However, this approach needs to be empirically tested in future research.

      CRediT authorship contribution statement

      Hannah Ewald: Writing – original draft, Writing – review & editing, Project administration. Irma Klerings: Conceptualization, Investigation, Writing – original draft, Writing – review & editing, Project administration. Gernot Wagner: Writing – review & editing. Thomas L. Heise: Writing – review & editing. Jan M. Stratil: Writing – review & editing. Stefan K. Lhachimi: Writing – review & editing. Lars G. Hemkens: Writing – review & editing. Gerald Gartlehner: Conceptualization, Writing – review & editing. Susan Armijo-Olivo: Writing – review & editing. Barbara Nussbaumer-Streit: Conceptualization, Writing – review & editing.

      Acknowledgments

      We thank everyone who was involved in the main project for their contributions indirectly help us realize this project.

      Appendix A. Supplementary Data

      References

        • Higgins J.P.T.
        • Thomas J.
        • Chandler J.
        • Cumpston M.
        • Li T.
        • Page M.J.
        • et al.
        Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021).
        (Available at)
        www.training.cochrane.org/handbook.2021
        Date: 2021
        Date accessed: January 20, 2022
        • Higgins J.P.T.
        • Lasserson T.
        • Chandler J.
        • Tovey D.
        • Thomas J.
        • Flemyng E.
        • et al.
        Methodological expectations of Cochrane intervention reviews.
        Cochrane, London2022
        • Hartling L.
        • Featherstone R.
        • Nuspl M.
        • Shave K.
        • Dryden D.M.
        • Vandermeer B.
        The contribution of databases to the results of systematic reviews: a cross-sectional study.
        BMC Med Res Methodol. 2016; 16: 127
        • Halladay C.W.
        • Trikalinos T.A.
        • Schmid I.T.
        • Schmid C.H.
        • Dahabreh I.J.
        Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions.
        J Clin Epidemiol. 2015; 68: 1076-1084
        • Frandsen T.F.
        • Eriksen M.B.
        • Hammer D.M.G.
        • Christensen J.B.
        PubMed coverage varied across specialties and over time: a large-scale study of included studies in Cochrane reviews.
        J Clin Epidemiol. 2019; 112: 59-66
        • Marshall I.J.
        • Marshall R.
        • Wallace B.C.
        • Brassey J.
        • Thomas J.
        Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study.
        J Clin Epidemiol. 2019; 109: 30-41
        • Aagaard T.
        • Lund H.
        • Juhl C.
        Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders?.
        BMC Med Res Methodol. 2016; 16: 161
        • Bramer W.M.
        • Giustini D.
        • Kramer B.M.
        Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study.
        Syst Rev. 2016; 5: 39
        • Bramer W.M.
        • Rethlefsen M.L.
        • Kleijnen J.
        • Franco O.H.
        Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study.
        Syst Rev. 2017; 6: 245
        • Levay P.
        • Raynor M.
        • Tuvey D.
        The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE public health guidance.
        Evid Based Libr Inf Pract. 2015; 10: 50-68
        • Nussbaumer-Streit B.
        • Klerings I.
        • Wagner G.
        • Heise T.L.
        • Dobrescu A.I.
        • Armijo-Olivo S.
        • et al.
        Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.
        J Clin Epidemiol. 2018; 102: 1-11
        • Nussbaumer-Streit B.
        • Klerings I.
        • Wagner G.
        • Titscher V.
        • Gartlehner G.
        Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study.
        Syst Rev. 2016; 5: 197
        • Ewald H.
        • Klerings I.
        • Wagner G.
        • Heise T.L.
        • Dobrescu A.I.
        • Armijo-Olivo S.
        • et al.
        Abbreviated and comprehensive literature searches led to identical or very similar effect estimates: a meta-epidemiological study.
        J Clin Epidemiol. 2020; 128: 1-12
        • Geretsegger M.
        • Elefant C.
        • Mössler Karin A.
        • Gold C.
        Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews.
        Cochrane Database Syst Rev. 2014; 2014: CD004381
        • McNamara D.A.
        • Goldberger J.J.
        • Berendsen M.A.
        • Huffman M.D.
        Implantable defibrillators versus medical therapy for cardiac channelopathies.
        Cochrane Database Syst Rev. 2015; 2015: CD011168
        • Wong G.W.
        • Wright J.M.
        Blood pressure lowering efficacy of nonselective beta-blockers for primary hypertension.
        Cochrane Database Syst Rev. 2014; 28: CD007452
        • Blessberger H.
        • Kammler J.
        • Domanovits H.
        • Schlager O.
        • Wildner B.
        • Azar D.
        • et al.
        Perioperative beta-blockers for preventing surgery-related mortality and morbidity.
        Cochrane Database Syst Rev. 2014; 2014: CD004476
        • Hooper L.
        • Martin N.
        • Abdelhamid A.
        • Davey Smith G.
        Reduction in saturated fat intake for cardiovascular disease.
        Cochrane Database Syst Rev. 2015; 5: CD011737
        • Hooper L.
        • Abdelhamid A.
        • Bunn D.
        • Brown T.
        • Summerbell C.D.
        • Skeaff C.M.
        Effects of total fat intake on body weight.
        Cochrane Database Syst Rev. 2015; : CD011834
        • Birks Jacqueline S.
        • Chong Lee Y.
        • Grimley Evans J.
        Rivastigmine for Alzheimer’s disease.
        Cochrane Database Syst Rev. 2015; 2015: CD001191
        • Minguet F.
        • Van Den Boogerd L.
        • Salgado T.M.
        • Correr C.J.
        • Fernandez-Llimos F.
        Characterization of the medical subject headings thesaurus for pharmacy.
        Am J Health Syst Pharm. 2014; 71: 1965-1972
        • Barroso J.
        • Gollop C.J.
        • Sandelowski M.
        • Meynell J.
        • Pearce P.F.
        • Collins L.J.
        The challenges of searching for and retrieving qualitative studies.
        West J Nurs Res. 2003; 25: 153-178
        • Kugley S.
        • Wade A.
        • Thomas J.
        • Mahood Q.
        • Jørgensen A.-M.K.
        • Hammerstrøm K.
        • et al.
        Searching for studies: a guide to information retrieval for Campbell systematic reviews.
        Campbell Syst Rev. 2017; 13: 1-73
        • Rethlefsen M.L.
        • Murad M.H.
        • Livingston E.H.
        Engaging medical librarians to improve the quality of review articles.
        JAMA. 2014; 312: 999-1000
        • Wong G.W.
        • Laugerotte A.
        • Wright J.M.
        Blood pressure lowering efficacy of dual alpha and beta blockers for primary hypertension.
        Cochrane Database Syst Rev. 2015; 2015: CD007449
        • Cooper C.
        • Booth A.
        • Britten N.
        • Garside R.
        A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
        Syst Rev. 2017; 6: 234
        • Vassar M.
        • Yerokhin V.
        • Sinnett P.M.
        • Weiher M.
        • Muckelrath H.
        • Carr B.
        • et al.
        Database selection in systematic reviews: an insight through clinical neurology.
        Health Inf Libr J. 2017; 34: 156-164
        • Hausner E.
        • Guddat C.
        • Hermanns T.
        • Lampert U.
        • Waffenschmidt S.
        Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one.
        J Clin Epidemiol. 2016; 77: 118-124
        • Bramer W.M.
        • de Jonge G.B.
        • Rethlefsen M.L.
        • Mast F.
        • Kleijnen J.
        A systematic approach to searching: an efficient and complete method to develop literature searches.
        J Med Libr Assoc. 2018; 106: 531-541
        • Hausner E.
        • Guddat C.
        • Hermanns T.
        • Lampert U.
        • Waffenschmidt S.
        Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
        J Clin Epidemiol. 2015; 68: 191-199
        • McGowan J.
        • Sampson M.
        • Salzwedel D.M.
        • Cogo E.
        • Foerster V.
        • Lefebvre C.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        J Clin Epidemiol. 2016; 75: 40-46
        • Franco J.V.A.
        • Garrote V.L.
        • Escobar Liquitay C.M.
        • Vietto V.
        Identification of problems in search strategies in Cochrane Reviews.
        Res Synth Methods. 2018; 9: 408-416