Journal of Clinical Epidemiology
Volume 63, Issue 10 , Pages 1061-1070 , October 2010

A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases

  • Tatyana Shamliyan

      Affiliations

    • Minnesota Evidence-based Practice Center, University of Minnesota School of Public Health, Minneapolis, MN 55455, USA
    • Corresponding Author InformationCorresponding author. University of Minnesota School of Public Health, D351 Mayo MMC 197, 420 Delaware St. SE, Minneapolis, MN 55455, USA. Tel. 612-624-1185.
    • ,
  • Robert L. Kane

      Affiliations

    • Minnesota Evidence-based Practice Center, University of Minnesota School of Public Health, Minneapolis, MN 55455, USA
    • ,
  • Stacy Dickinson

      Affiliations

    • University of Minnesota School of Public Health, Minneapolis, MN 55455, USA

,Accepted 4 April 2010.

References 

  1. Fox DM. Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Aff (Millwood). 2005;24:114–122
  2. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;(10 Suppl 1):35–48
  3. Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997;127(1):37–42
  4. Chan KS, Morton SC, Shekelle PG. Systematic reviews for evidence-based management: how to find them and what to do with them. Am J Manag Care. 2004;10(11 Pt 1):806–812
  5. Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, Truman BI, et al. Developing an evidence-based Guide to Community Preventive Services—methods. The Task Force on Community Preventive Services. Am J Prev Med. 2000;18(1 Suppl):35–43
  6. Agency for Healthcare Research and Quality . Systems to rate the strength of scientific evidence. Rockville, MD: University of California-San Francisco-Stanford Evidence-Based Practice Center; 2002;
  7. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36:666–676
  8. National Center for Biotechnology Information, U.S. National Library of Medicine. Bethesda, MD. Available at: http://www.ncbi.nlm.nih.gov/pubmed. Accessed June 2010.
  9. The Cochrane database of systematic reviews. 2000-2010. New York: Wiley.
  10. OCLC . WorldCat. Dublin, OH: OCLC; 2001;
  11. Elsevier Science (Firm) . Scirus for scientific information only. Amsterdam: Elsevier Science; 2001;
  12. Higgins J, Green S Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Chichester, West Sussex; Hoboken NJ: John Wiley & Sons; 2008;
  13. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453–1457
  14. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–2012
  15. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62:1013–1020
  16. Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. Lancet. 2008;371(9619):1149–1150
  17. Higgins J, Green S The Cochrane Collaboration. The Cochrane handbook for systematic reviews of interventions. Available at http://www2.cochrane.org/resources/handbook/handbook.pdf2005;Accessed June 2010
  18. Aschengrau A, Seage GR. Essentials of Epidemiology in Public Health. Sudbury, MA: Jones and Bartlett; 2003;
  19. BMJ Publishing Group . Clinical evidence. Available at http://www.clinicalevidence.com2005;Accessed June 2010
  20. Hulley SB. Designing clinical research: an epidemiologic approach. 2nd edition. Philadelphia, PA: Lippincott Williams & Wilkins; 2001;
  21. Centers for Disease Control and Prevention (U.S.) , Career Development Division . Principles of epidemiology in public health practice: an introduction to applied epidemiology and biostatistics. 3rd edition. Atlanta, GA: U.S. Dept. of Health and Human Services, Centers for Disease Control and Prevention (CDC), Office of Workforce and Career Development; 2006;
  22. Stata Corporation . Stata base reference manual. Release 10 edition. College Station, TX: Stata Corporation; 2007;
  23. SAS Institute . SAS com. Cary, NC: SAS Institute; 2001;
  24. Loney PL, Chambers LW, Bennett KJ, Roberts JG, Stratford PW. Critical appraisal of the health research literature: prevalence or incidence of a health problem. Chronic Dis Can. 1998;19(4):170–176
  25. DuRant RH. Checklist for the evaluation of research articles. J Adolesc Health. 1994;15(1):4–8
  26. Woodbury MG, Houghton PE. Prevalence of pressure ulcers in Canadian healthcare settings. Ostomy Wound Manage. 2004;50(10):22–246, 8, 30, 2, 4, 6–8
  27. Macfarlane TV, Glenny AM, Worthington HV. Systematic review of population-based epidemiological studies of oro-facial pain. J Dent. 2001;29(7):451–467
  28. Steinberg EP, Eknoyan G, Levin NW, Eschbach JW, Golper TA, Owen WF, et al. Methods used to evaluate the quality of evidence underlying the National Kidney Foundation-Dialysis Outcomes Quality Initiative Clinical Practice Guidelines: description, findings, and implications. Am J Kidney Dis. 2000;36(1):1–11
  29. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–1278
  30. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377–384
  31. Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J. Methodological index for non-randomized studies (MINORS): development and validation of a new instrument. ANZ J Surg. 2003;73(9):712–716
  32. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52(7):625–629
  33. Horwitz RI, Feinstein AR. Methodologic standards and contradictory results in case-control research. Am J Med. 1979;66:556–564
  34. Lichtenstein MJ, Mulrow CD, Elwood PC. Guidelines for reading case-control studies. J Chronic Dis. 1987;40:893–903
  35. Zaza S, Wright-De Aguero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, et al. Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med. 2000;18(1 Suppl):44–74
  36. Spitzer WO, Lawrence V, Dales R, Hill G, Archer MC, Clark P, et al. Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking. Clin Invest Med. 1990;13(1):17–42discussion 3–6
  37. Krogh CL. A checklist system for critical review of medical literature. Med Educ. 1985;19(5):392–395
  38. Cho MK, Bero LA. Instruments for assessing the quality of drug studies published in the medical literature. JAMA. 1994;272(2):101–104
  39. Gardner MJ, Machin D, Campbell MJ. Use of check lists in assessing the statistical content of medical studies. Br Med J (Clin Res Ed). 1986;292(6523):810–812
  40. Esdaile JM, Horwitz RI. Observational studies of cause-effect relationships: an analysis of methodologic problems as illustrated by the conflicting data for the role of oral contraceptives in the etiology of rheumatoid arthritis. J Chronic Dis. 1986;39(10):841–852
  41. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121(1):11–21
  42. Cowley DE. Prostheses for primary total hip replacement. A critical appraisal of the literature. Int J Technol Assess Health Care. 1995 Fall;11:770–778
  43. Angelillo IF, Villari P. Residential exposure to electromagnetic fields and childhood leukaemia: a meta-analysis. Bull World Health Organ. 1999;77(11):906–915
  44. Nguyen QV, Bezemer PD, Habets L, Prahl-Andersen B. A systematic review of the relationship between overjet size and traumatic dental injuries. Eur J Orthod. 1999;21:503–515
  45. van der Windt DA, Thomas E, Pope DP, de Winter AF, Macfarlane GJ, Bouter LM, et al. Occupational risk factors for shoulder pain: a systematic review. Occup Environ Med. 2000;57(7):433–442
  46. Borghouts JA, Koes BW, Bouter LM. The clinical course and prognostic factors of non-specific neck pain: a systematic review. Pain. 1998;77(1):1–13
  47. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(3 Suppl):21–35
  48. Reisch JS, Tyson JE, Mize SG. Aid to the evaluation of therapeutic studies. Pediatrics. 1989;84:815–827
  49. Al-Jader LN, Newcombe RG, Hayes S, Murray A, Layzell J, Harper PS. Developing a quality scoring system for epidemiological surveys of genetic disorders. Clin Genet. 2002;62(3):230–234
  50. Scholten-Peeters GGM, Verhagen AP, Bekkering GE, van der Windt DAWM, Barnsley L, Oostendorp RAB, et al. Prognostic factors of whiplash-associated disorders: a systematic review of prospective cohort studies. Pain. 2003;104:303–322
  51. Rangel SJ, Kelsey J, Colby CE, Anderson J, Moss RL. Development of a quality assessment scale for retrospective clinical studies in pediatric surgery. J Pediatr Surg. 2003;38:390–396discussion 390–6
  52. Garber BG, Hebert PC, Yelle JD, Hodder RV, McGowan J. Adult respiratory distress syndrome: a systemic overview of incidence and risk factors. Crit Care Med. 1996;24:687–695
  53. de Boer AG, Verbeek JH, van Dijk FJ. Adult survivors of childhood cancer and unemployment: a metaanalysis. Cancer. 2006;107(1):1–11
  54. Anders JF, Jacobson RM, Poland GA, Jacobsen SJ, Wollan PC. Secondary failure rates of measles vaccines: a metaanalysis of published studies. Pediatr Infect Dis J. 1996;15(1):62–66
  55. Manchikanti L, Singh V, Vilims BD, Hansen HC, Schultz DM, Kloth DS. Medial branch neurotomy in management of chronic spinal pain: systematic review of the evidence. Pain Physician. 2002;5:405–418
  56. Jabbour M, Osmond MH, Klassen TP. Life support courses: are they effective?. Ann Emerg Med. 1996;28:690–698
  57. Shea B, Boers M, Grimshaw JM, Hamel C, Bouter LM. Does updating improve the methodological and reporting quality of systematic reviews?. BMC Med Res Methodol. 2006;6:27
  58. Ariens GA, van Mechelen W, Bongers PM, Bouter LM, van der Wal G. Physical risk factors for neck pain. Scand J Work Environ Health. 2000;26(1):7–19
  59. Longnecker MP, Berlin JA, Orza MJ, Chalmers TC. A meta-analysis of alcohol consumption in relation to risk of breast cancer. JAMA. 1988;260:652–656
  60. Corrao G, Bagnardi V, Zambon A, Arico S. Exploring the dose-response relationship between alcohol consumption and the risk of several alcohol-related conditions: a meta-analysis. Addiction. 1999;94:1551–1573
  61. Stock SR. Workplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: a meta-analysis. Am J Ind Med. 1991;19(1):87–107
  62. Berlin JA, Colditz GA. A meta-analysis of physical activity in the prevention of coronary heart disease. Am J Epidemiol. 1990;132:612–628
  63. Kreulen CM, Creugers NH, Meijering AC. Meta-analysis of anterior veneer restorations in clinical studies. J Dent. 1998;26(4):345–353
  64. Littenberg B, Weinstein LP, McCarren M, Mead T, Swiontkowski MF, Rudicel SA, et al. Closed fractures of the tibial shaft. A meta-analysis of three methods of treatment. J Bone Joint Surg Am. 1998;80(2):174–183
  65. Campos-Outcalt D, Senf J, Watkins AJ, Bastacky S. The effects of medical school curricula, faculty role models, and biomedical research support on choice of generalist physician careers: a review and quality assessment of the literature. Acad Med. 1995;70(7):611–619
  66. Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–434
  67. Spencer-Green G. Outcomes in primary Raynaud phenomenon: a meta-analysis of the frequency, rates, and predictors of transition to secondary diseases. Arch Intern Med. 1998;158(6):595–600
  68. Meijer R, Ihnenfeldt DS, van Limbeek J, Vermeulen M, de Haan RJ. Prognostic factors in the subacute phase after stroke for the future residence after six months to one year. A systematic review of the literature. Clin Rehabil. 2003;17:512–520
  69. Bornhoft G, Maxion-Bergemann S, Wolf U, Kienle GS, Michalsen A, Vollmar HC, et al. Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity. BMC Med Res Methodol. 2006;6:56
  70. Cameron I, Crotty M, Currie C, Finnegan T, Gillespie L, Gillespie W, et al. Geriatric rehabilitation following fractures in older people: a systematic review. Health Technol Assess. 2000;4(2):i–iv, 1–111
  71. Hadorn DC, Baker D, Hodges JS, Hicks N. Rating the quality of evidence for clinical practice guidelines. J Clin Epidemiol. 1996;49:749–754
  72. Conde-Agudelo A, Villar J, Lindheimer M. Maternal infection and risk of preeclampsia: systematic review and metaanalysis. Am J Obstet Gynecol. 2008;198(1):7–22
  73. Fowkes FG, Fulton PM. Critical appraisal of published research: introductory guidelines. BMJ (Clin Res Ed). 1991;302(6785):1136–1140
  74. Scottish Intercollegiate Guidelines Network , Harbour RT, Forsyth L. Sign 50. A guideline developer's handbook. Revised edition. Ediburgh, Scotland: Scottish Intercollegiate Guidelines Network; 2008;1 text file (102 p)
  75. Pilote L, Tager IB. Outcomes research in the development and evaluation of practice guidelines. BMC Health Serv Res. 2002;2(1):7
  76. Hirtz D, Thurman DJ, Gwinn-Hardy K, Mohamed M, Chaudhuri AR, Zalutsky R. How common are the “common” neurologic disorders?. Neurology. 2007;68(5):326–337
  77. Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V. Users' guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group. JAMA. 1994;271(20):1615–1619
  78. Elwood M. Forward projection–using critical appraisal in the design of studies. Int J Epidemiol. 2002;31:1071–1073
  79. Pavia M, Pileggi C, Nobile CG, Angelillo IF. Association between fruit and vegetable consumption and oral cancer: a meta-analysis of observational studies. Am J Clin Nutr. 2006;83:1126–1134
  80. Genaidy AM, Lemasters GK, Lockey J, Succop P, Deddens J, Sobeih T, et al. An epidemiological appraisal instrument—a tool for evaluation of epidemiological studies. Ergonomics. 2007;50:920–960
  81. Bhutta AT, Cleves MA, Casey PH, Cradock MM, Anand KJ. Cognitive and behavioral outcomes of school-aged children who were born preterm: a meta-analysis. JAMA. 2002;288:728–737
  82. Dixon-Woods M, Sutton A, Shaw R, Miller T, Smith J, Young B, et al. Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. J Health Serv Res Policy. 2007;12(1):42–47
  83. Treadwell JR, Tregear SJ, Reston JT, Turkelson CM. A system for rating the stability and strength of medical evidence. BMC Med Res Methodol. 2006;6:52
  84. Groenwold RH, Van Deursen AM, Hoes AW, Hak E. Poor quality of reporting confounding bias in observational intervention studies: a systematic review. Ann Epidemiol. 2008;18(10):746–751
  85. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ (Clin Res Ed). 2004;328(7430):22–24
  86. Ezzati M, Hoorn SV, Rodgers A, Lopez AD, Mathers CD, Murray CJ. Estimates of global and regional potential health gains from reducing multiple major risk factors. Lancet. 2003;362(9380):271–280
  87. Arnold PD, Ickowicz A, Chen S, Schachar R. Attention-deficit hyperactivity disorder with and without obsessive-compulsive behaviours: clinical characteristics, cognitive assessment, and risk factors. Can J Psychiatry. 2005;50(1):59–66
  88. Lindsberg PJ, Grau AJ. Inflammation and infections as risk factors for ischemic stroke. Stroke. 2003;34:2518–2532
  89. BeLue R, Taylor-Richardson KD, Lin J, Rivera AT, Grandison D. African Americans and participation in clinical trials: differences in beliefs and attitudes by gender. Contemp Clin Trials. 2006;27:498–505
  90. Braunstein JB, Sherber NS, Schulman SP, Ding EL, Powe NR. Race, medical researcher distrust, perceived harm, and willingness to participate in cardiovascular prevention trials. Medicine (Baltimore). 2008;87(1):1–9
  91. Smith YR, Johnson AM, Newman LA, Greene A, Johnson TR, Rogers JL. Perceptions of clinical research participation among African American women. J Womens Health (Larchmt). 2007;16:423–428
  92. Hussain-Gambles M, Leese B, Atkin K, Brown J, Mason S, Tovey P. Involving South Asian patients in clinical trials. Health Technol Assess (Winchester, Engl). 2004;8(42):iii, 1–109
  93. Corbie-Smith G, Thomas SB, Williams MV, Moody-Ayers S. Attitudes and beliefs of African Americans toward participation in medical research. J Gen Intern Med. 1999;14(9):537–546
  94. Linden HM, Reisch LM, Hart A, Harrington MA, Nakano C, Jackson JC, et al. Attitudes toward participation in breast cancer randomized clinical trials in the African American community: a focus group study. Cancer Nurs. 2007;30(4):261–269
  95. Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006;59:1249–1256
  96. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ (Clin Res Ed). 2005;330(7499):1053
  97. Greenland S, O'Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2:463–471
  98. Whiting P, Harbord R, Kleijnen J. No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Med Res Methodol. 2005;5:19

PII: S0895-4356(10)00176-9

doi: 10.1016/j.jclinepi.2010.04.014

Journal of Clinical Epidemiology
Volume 63, Issue 10 , Pages 1061-1070 , October 2010

Article Tools

* Image Usage & Resolution