Advertisement
AHRQ Series Part II: Methods Guide for Comparative Effectiveness - Guest Editor, Mark Helfand| Volume 64, ISSUE 11, P1168-1177, November 2011

Download started.

Ok

Finding evidence for comparing medical interventions: AHRQ and the Effective Health Care Program

  • Rose Relevo
    Correspondence
    Corresponding author. Tel.: +503-220-8262 ×51318; fax: +503-494-4551.
    Affiliations
    Scientific Resource Center, AHRQ Effective Health Care Program, Oregon Evidence-based Practice Center, Oregon Health and Science University, 3181 SW Sam Jackson Park Road, Mail code: BICC, Portland, OR 97239-3098, USA
    Search for articles by this author
  • Howard Balshem
    Affiliations
    Scientific Resource Center, AHRQ Effective Health Care Program, Oregon Evidence-based Practice Center, Oregon Health and Science University, 3181 SW Sam Jackson Park Road, Mail code: BICC, Portland, OR 97239-3098, USA
    Search for articles by this author

      Abstract

      Objective

      This article discusses search methodology to identify evidence for comparative effectiveness reviews (CERs) as practiced by the Effective Health Care program.

      Study Design and Setting

      Review.

      Results

      Search methods described attempt to overcome the bias inherent in the publication and distribution of clinical evidence. Bibliographic databases and search strategies are discussed with special emphasis on searching for observational studies and harms data. Other techniques described include the use of key articles, citation tracking, hand searching, and personal communications. Strategies for locating gray literature, such as clinical trial protocols and regulatory information, are described. Search reporting and other practical matters are also discussed.

      Conclusion

      Better reporting and further research on search strategies is needed to develop additional evidence-based recommendations.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Whitlock E.P.
        • Lin J.S.
        • Chou R.
        • Shekelle P.
        • Robinson K.A.
        Using existing systematic reviews in complex systematic reviews.
        Ann Intern Med. 2008; 148: 776-782
        • Whitlock E.P.
        • Lopez S.A.
        • Chang S.
        • Helfand M.
        • Eder M.
        • Floyd N.
        AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health care program.
        J Clin Epidemiol. 2010; 63: 491-501
        • Medical Library Association
        Role of expert searching in health sciences libraries: policy statement by the Medical Library Association adopted September 2003.
        J Med Libr Assoc. 2005; 93: 42-44
        • McGowan J.
        • Sampson M.
        Systematic reviews need systematic searchers.
        J Med Libr Assoc. 2005; 93: 74-80
        • Golder S.
        • Loke Y.
        • McIntosh H.M.
        Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects.
        J Clin Epidemiol. 2008; 61: 440-448
        • Mokkink L.B.
        • Terwee C.B.
        • Stratford P.W.
        • Alonso J.
        • Patrick D.L.
        • Riphagen I.
        • et al.
        Evaluation of the methodological quality of systematic reviews of health status measurement instruments.
        Qual Life Res. 2009; 18: 313-333
        • Alberani V.
        • De Castro Pietrangeli P.
        • Mazza A.M.
        The use of grey literature in health sciences: a preliminary survey.
        Bull Med Libr Assoc. 1990; 78: 358-363
        • Illig J.
        Archiving “event knowledge”: bringing “dark data” to light.
        J Med Libr Assoc. 2008; 96: 189-191
      1. Grey Literature Network Service, editor. New frontiers in grey literature. Fourth International conference on Grey Literature. Washington, DC: GL’99 proceedings; October 4–5, 1999.

        • Conn V.S.
        • Valentine J.C.
        • Cooper H.M.
        • Rantz M.J.
        Grey literature in meta-analyses.
        Nurs Res. 2003; 52 ([Review]): 256-261
        • Dickersin K.
        • Scherer R.
        • Lefebvre C.
        Identifying relevant studies for systematic reviews.
        BMJ. 1994; 309 ([Review]): 1286-1291
        • McAuley L.
        • Pham B.
        • Tugwell P.
        • Moher D.
        Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?.
        Lancet. 2000; 356: 1228-1231
        • Hopewell S.
        • McDonald S.
        • Clarke Mike J.
        • Egger M.
        Grey literature in meta-analyses of randomized trials of health care interventions.
        Cochrane Database Syst Rev [Serial on the Internet]. 2007; 2 (Available at) (Accessed April 28, 2011)
        • Hartling L.
        • McAlister F.A.
        • Rowe B.H.
        • Ezekowitz J.
        • Friesen C.
        • Klassen T.P.
        Challenges in systematic reviews of therapeutic devices and procedures.
        Ann Intern Med. 2005; 142: 1100-1111
        • Helmer D.
        • Savoie I.
        • Green C.
        • Kazanjian A.
        Evidence-based practice: extending the search to find material for the systematic review.
        Bull Med Libr Assoc. 2001; 89: 346-352
        • Shekelle P.G.
        • Morton S.C.
        • Suttorp M.J.
        • Buscemi N.
        • Friesen C.
        Challenges in systematic reviews of complementary and alternative medicine topics.
        Ann Intern Med. 2005; 142: 1042-1047
        • Egger M.
        • Juni P.
        • Bartlett C.
        • Holenstein F.
        • Sterne J.
        How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
        Health Technol Asses. 2003; 7: 1-76
        • Cook A.M.
        • Finlay I.G.
        • Edwards A.G.
        • Hood K.
        • Higginson I.J.
        • Goodwin D.M.
        • et al.
        Efficiency of searching the grey literature in palliative care.
        J Pain Symptom Manage. 2001; 22: 797-801
        • Fergusson D.
        • Laupacis A.
        • Salmi L.R.
        • McAlister F.A.
        • Huet C.
        What should be included in meta-analyses? An exploration of methodological issues using the ISPOT meta-analyses.
        Int J Technol Assess Health Care. 2000; 16: 1109-1119
        • van Driel M.L.
        • De Sutter A.
        • De Maeseneer J.
        • Christiaens T.
        Searching for unpublished trials in Cochrane reviews may not be worth the effort.
        J Clin Epidemiol. 2009; 62 (e3): 838-844
        • Bennett D.A.
        • Jull A.
        FDA: untapped source of unpublished trials.
        Lancet. 2003; 361: 1402-1403
        • MacLean C.H.
        • Morton S.C.
        • Ofman J.J.
        • Roth E.A.
        • Shekelle P.G.
        How useful are unpublished data from the Food and Drug Administration in meta-analysis?.
        J Clin Epidemiol. 2003; 56: 44-51
        • Man-Son-Hing M.
        • Wells G.
        • Lau A.
        Quinine for nocturnal leg cramps: a meta-analysis including unpublished data.
        J Gen Intern Med. 1998; 13: 600-606
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective publication of antidepressant trials and its influence on apparent efficacy.
        N Engl J Med. 2008; 358: 252-260
        • Health Canada Drug Products Database
        Health Canada.
        (Available at) (Accessed September 21, 2010)
        • European Public Assessement Reports
        European Medicines Agency.
        (Available at) (Accessed September 21, 2010)
        • Savoie I.
        • Helmer D.
        • Green C.J.
        • Kazanjian A.
        Beyond MEDLINE: reducing bias through extended systematic review search.
        Int J Technol Assess Health Care. 2003; 19 ([Journal Article]): 168-178
        • Mathieu S.
        • Boutron I.
        • Moher D.
        • Altman D.G.
        • Ravaud P.
        Comparison of registered and published primary outcomes in randomized controlled trials.
        JAMA. 2009; 302: 977-984
      2. U.S. National Institutes of Health. ClinicalTrials.gov. 2010; Available at http://clinicaltrials.gov. Accessed September 21, 2010.

        • Current Controlled Trials
        BioMed Central.
        (Available at) (Accessed September 21, 2010)
      3. Clinical Study Results. Available at http://www.clinicalstudyresults.org/home/. Accessed September 21, 2010.

        • WHO International Clinical Trials Registry Platform
        World Health Organization.
        (Available at) (Accessed September 21, 2010)
        • von Elm E.
        • Costanza M.C.
        • Walder B.
        • Tramer M.R.
        More insight into the fate of biomedical meeting abstracts: a systematic review.
        BMC Med Res Methodol. 2003; 3: 12
        • Toma M.
        • McAlister F.A.
        • Bialy L.
        • Adams D.
        • Vandermeer B.
        • Armstrong P.W.
        Transition from meeting abstract to full-length journal article for randomized controlled trials.
        JAMA. 2006; 295: 1281-1287
        • Matthews E.J.
        • Edwards A.G.
        • Barker J.
        • Bloor M.
        • Covey J.
        • Hood K.
        • et al.
        Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care.
        Health Libr Rev. 1999; 16: 112-120
        • Brettle A.J.
        • Long A.F.
        Comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness.
        Bull Med Libr Assoc. 2001; 89: 353-362
        • O’Leary N.
        • Tiernan E.
        • Walsh D.
        • Lucey N.
        • Kirkova J.
        • Davis M.P.
        The pitfalls of a systematic MEDLINE review in palliative medicine: symptom assessment instruments.
        Am J Hosp Palliat Care. 2007; 24: 181-184
        • Glanville J.M.
        • Lefebvre C.
        • Miles J.N.
        • Camosso-Stefinovic J.
        How to identify randomized controlled trials in MEDLINE: ten years on.
        J Med Libr Assoc. 2006; 94: 130-136
        • McKibbon K.A.
        • Wilczynski N.L.
        • Haynes R.B.
        • Hedges T.
        Retrieving randomized controlled trials from MEDLINE: a comparison of 38 published search filters.
        Health Info Libr J. 2009; 26: 187-202
        • Haynes R.B.
        • Wilczynski N.
        • McKibbon K.A.
        • Walker C.J.
        • Sinclair J.C.
        Developing optimal search strategies for detecting clinically sound studies in MEDLINE.
        J Am Med Inform Assoc. 1994; 1: 447-458
      4. Cochrane Handbook for Systematic Reviews of Interventions: 6.4.11 search filters 2008: Version 5.0.1.
        (Available at) (Updated September 2008. Accessed September 21, 2010)
        • Scottish Intercollegiate Guidelines Network (SIGN)
        Search filters. Edingurgh2009.
        (Available at) (Updated August 3, 2009. Accessed September 21, 2010)
        • InterTASC Information Specialists’ Sub-Group
        Search Filter Resource 2009.
        (Available at) (Updated July 2, 2009. Accessed September 21, 2010)
        • Bak G.
        • Mierzwinski-Urban M.
        • Fitzsimmons H.
        • Morrison A.
        • Maden-Jenkins M.
        A pragmatic critical appraisal instrument for search filters: introducing the CADTH CAI.
        Health Info Libr J. 2009; 26: 211-219
        • Sampson M.
        • McGowan J.
        • Lefebvre C.
        • Moher D.
        • Grimshaw J.M.
        PRESS: peer review of electronic search strategies.
        Canadian Agency for Drugs and Technologies in Health, Ottowa, Canada2008
        • Conn V.S.
        • Isaramalai S.A.
        • Rath S.
        • Jantarakupt P.
        • Wadhawan R.
        • Dash Y.
        Beyond MEDLINE for literature searches.
        J Nurs Scholarsh. 2003; 35 ([Review]): 177-182
        • Morrison A.
        • Moulton K.
        • Clark M.
        • Polisena J.
        • Fiander M.
        • Mierzwinski-Urban M.
        • et al.
        English-language restriction when conducting systematic review-based meta-analyses: systematic review of published studies.
        ([Serial on the Internet]) Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada2009 (Available at) (Accessed April 28, 2011)
        • Hernandez D.A.
        • El-Masri M.M.
        • Hernandez C.A.
        Choosing and using citation and bibliographic database software (BDS).
        Diabetes Educ. 2008; 34: 457-474
        • Gomis M.
        • Gall C.
        • Brahmi F.A.
        Web-based citation management compared to end note: options for medical sciences.
        Med Ref Serv Q. 2008; 27: 260-271
      5. White CM, Ip S, McPheeters M, Carey TS, Cour R, Lohr KN. Using existing systematic reviews to replace de novo processes in conducting Comparative Effectiveness Reviews. In: Agency for Healthcare Research and Quality. Methods Guide for Comparative Effectiveness Reviews [posted September 2009]. Rockville, MD. Available at http://effectivehealthcare.ahrq.gov/healthInfo.cfm?infotype=rr&ProcessID=60. Accessed September 21, 2010.

        • Loke Y.K.
        • Price D.
        • Herxheimer A.
        • Group C.A.E.M.
        Systematic reviews of adverse effects: framework for a structured approach.
        BMC Med Res Methodol. 2007; 7: 32
        • Chou R.
        • Helfand M.
        Challenges in systematic reviews that assess treatment harms.
        Ann Intern Med. 2005; 142 ([Review]): 1090-1099
        • Chou R.
        • Aronson N.
        • Atkins D.
        • Ismaila A.S.
        • Santaguida P.
        • Smith D.H.
        • et al.
        AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program.
        J Clin Epidemiol. 2010; 63: 502-512
        • Derry S.
        • Kong Loke Y.
        • Aronson J.K.
        Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials.
        BMC Med Res Methodol. 2001; 1: 7
        • Golder S.
        • McIntosh H.M.
        • Duffy S.
        • Glanville J.
        Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE.
        Health Info Libr J. 2006; 23: 3-12
        • Wieland S.
        • Dickersin K.
        Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives.
        J Clin Epidemiol. 2005; 58: 560-567
        • Loke Y.K.
        • Price D.
        • Herxheimer A.
        Appendix 6b. Including adverse effects.
        in: Higgins J.P. Green S. Cochrane handbook for systematic reviews of interventions 425 [Updated May 2005]. Cochrane Colloboration; Cochrane Adverse Effects Subgroup, Chichester, UK2007: 194-195
        • Fraser C.
        • Murray A.
        • Burr J.
        Identifying observational studies of surgical interventions in MEDLINE and EMBASE.
        BMC Med Res Methodol. 2006; 6: 41
        • von Elm E.
        • Altman D.G.
        • Egger M.
        • Pocock S.J.
        • Gotzsche P.C.
        • Vandenbroucke J.P.
        • et al.
        The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies.
        Ann Intern Med. 2007; 147: 573-577
        • Golder S.
        • Loke Y.
        • McIntosh H.M.
        Room for improvement? A survey of the methods used in systematic reviews of adverse effects.
        BMC Med Res Methodol. 2006; 6: 3
        • Furlan A.D.
        • Irvin E.
        • Bombardier C.
        Limited search strategies were effective in finding relevant nonrandomized studies.
        J Clin Epidemiol. 2006; 59: 1303-1311
        • Scottish Intercollegiate Guidelines Network
        Search filters.
        (Available at) (Accessed September 21, 2010)
        • Vandenbroucke J.P.
        • von Elm E.
        • Altman D.G.
        • Gotzsche P.C.
        • Mulrow C.D.
        • Pocock S.J.
        • et al.
        Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration.
        Ann Intern Med. 2007; 147 ([Web only]): W163-W194
        • Zheng M.H.
        • Zhang X.
        • Ye Q.
        • Chen Y.P.
        Searching additional databases except PubMed are necessary for a systematic review.
        Stroke. 2008; 39 (author reply e40): e139
        • Suarez-Almazor M.E.
        • Belseck E.
        • Homik J.
        • Dorgan M.
        • Ramos-Remus C.
        Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough.
        Control Clin Trials. 2000; 21: 476-487
        • Betran A.P.
        • Say L.
        • Gulmezoglu A.M.
        • Allen T.
        • Hampson L.
        Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality.
        BMC Med Res Methodol. 2005; 5: 6
        • Sampson M.
        • Barrowman N.J.
        • Moher D.
        • Klassen T.P.
        • Pham B.
        • Platt R.
        • et al.
        Should meta-analysts search Embase in addition to Medline?.
        J Clin Epidemiol. 2003; 56: 943-955
        • DeLuca J.B.
        • Mullins M.M.
        • Lyles C.M.
        • Crepaz N.
        • Kay L.
        • Thadiparthi S.
        Developing a comprehensive search strategy for evidence based systematic reviews.
        Evid Based Libr Inf Pract. 2008; 3: 3-32
        • Kuper H.
        • Nicholson A.
        • Hemingway H.
        Searching for observational studies: what does citation tracking add to PubMed? A case study in depression and coronary heart disease.
        BMC Med Res Methodol. 2006; 6: 4
      6. Google Scholar. Available at http://scholar.google.com/. Accessed September 21, 2010.

        • Scopus
        Elsevier.
        (Available at) (Accessed September 21, 2010)
      7. PubFocus. Available at http://pubfocus.com/. Accessed September 21, 2010.

        • PubMed
        PubReMiner.
        (Available at) (Accessed September 21, 2010)
        • Falagas M.E.
        • Pitsouni E.I.
        • Malietzis G.A.
        • Pappas G.
        Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses.
        FASEB J. 2008; 22: 338-342
        • Salisbury L.
        Web of Science and Scopus: a comparative review of content and searching capabilities.
        The Charleston Advisor. 2009; 11: 5-18
        • Jasco P.
        As we may search—comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases.
        Curr Sci. 2005; 89: 1537-1547
        • Bakkalbasi N.
        • Bauer K.
        • Glover J.
        • Wang L.
        Three options for citation tracking: Google Scholar, Scopus and Web of Science.
        Biomed Digit Libr. 2006; 3: 7
        • Jadad A.R.
        • McQuay H.J.
        Searching the literature. Be systematic in your searching.
        BMJ. 1993; 307 ([Comment]): 66
        • Gotzsche P.C.
        Reference bias in reports of drug trials.
        Br Med J (Clin Res Ed). 1987; 295: 654-656
      8. Armour T, Dingwall O, Sampson M. Contribution of checking reference lists to systematic reviews. Poster presentation at: XIII Cochrane Colloquium. Melbourne: Australia; 2005.

        • Al Hajeri A.
        • Al Sayyad J.
        • Eisinga A.
        Handsearching the EMHJ for reports of randomized controlled trials by U.K. Cochrane Centre (Bahrain).
        East Mediterr Health J. 2006; 12: S253-S257
        • Jadad A.R.
        • Moher D.
        • Klassen T.P.
        Guides for reading and interpreting systematic reviews: II. How did the authors find the studies and assess their quality?.
        Arch Pediatr Adolesc Med. 1998; 152: 812-817
        • Hopewell S.
        • Clarke M.
        • Lusher A.
        • Lefebvre C.
        • Westby M.
        A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials.
        Stat Med. 2002; 21: 1625-1634
        • Avenell A.
        • Handoll H.H.
        • Grant A.M.
        Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture.
        Am J Clin Nutr. 2001; 73: 505-510
        • Armstrong R.
        • Jackson N.
        • Doyle J.
        • Waters E.
        • Howes F.
        It’s in your hands: the value of handsearching in conducting systematic reviews of public health interventions.
        J Public Health (Oxf). 2005; 27: 388-391
        • Zarin D.A.
        • Ide N.C.
        • Tse T.
        • Harlan W.R.
        • West J.C.
        • Lindberg D.A.
        Issues in the registration of clinical trials.
        JAMA. 2007; 297: 2112-2120
        • Tramer M.R.
        • Reynolds D.J.
        • Moore R.A.
        • McQuay H.J.
        Impact of covert duplicate publication on meta-analysis: a case study.
        BMJ. 1997; 315: 635-640
        • Reveiz L.
        • Cardona A.F.
        • Ospina E.G.
        • de Agular S.
        An e-mail survey identified unpublished studies for systematic reviews.
        J Clin Epidemiol. 2006; 59: 755-758
        • Kelley G.A.
        • Kelley K.S.
        • Tran Z.V.
        Retrieval of missing data for meta-analysis: a practical example.
        Int J Technol Assess Health Care. 2004; 20: 296-299
        • Peinemann F.
        • McGauran N.
        • Sauerland S.
        • Lange S.
        Negative pressure wound therapy: potential publication bias caused by lack of access to unpublished study results data.
        BMC Med Res Methodol. 2008; 8: 4
        • Rennie D.
        Fair conduct and fair reporting of clinical trials.
        JAMA. 1999; 282: 1766-1768
        • Gibson C.A.
        • Bailey B.W.
        • Carper M.J.
        • Lecheminant J.D.
        • Kirk E.P.
        • Huang G.
        • et al.
        Author contacts for retrieval of data for a meta-analysis on exercise and diet restriction.
        Int J Technol Assess Health Care. 2006; 22: 267-270
        • Yoshii A.
        • Plaut D.A.
        • McGraw K.A.
        • Anderson M.J.
        • Wellik K.E.
        Analysis of the reporting of search strategies in Cochrane systematic reviews.
        J Med Libr Assoc. 2009; 97: 21-29
        • Sampson M.
        • McGowan J.
        • Tetzlaff J.
        • Cogo E.
        • Moher D.
        No consensus exists on search reporting methods for systematic reviews.
        J Clin Epidemiol. 2008; 61: 748-754
        • Egger M.
        • Juni P.
        • Bartlett C.
        • CONSORT Group
        Value of flow diagrams in reports of randomized controlled trials.
        JAMA. 2001; 285: 1996-1999
        • Hopewell S.
        • Clarke M.
        • Moher D.
        • Wager E.
        • Middleton P.
        • Altman D.G.
        • et al.
        CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.
        PLoS Med. 2008; 5: e20
        • Spoor P.
        • Airey M.
        • Bennett C.
        • Greensill J.
        • Williams R.
        Use of the capture-recapture technique to evaluate the completeness of systematic literature searches.
        BMJ. 1996; 313: 342-343