AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews

Published:February 23, 2009DOI:https://doi.org/10.1016/j.jclinepi.2008.10.009

      Abstract

      Objective

      Our purpose was to measure the agreement, reliability, construct validity, and feasibility of a measurement tool to assess systematic reviews (AMSTAR).

      Study Design and Setting

      We randomly selected 30 systematic reviews from a database. Each was assessed by two reviewers using: (1) the enhanced quality assessment questionnaire (Overview of Quality Assessment Questionnaire [OQAQ]); (2) Sacks' instrument; and (3) our newly developed measurement tool (AMSTAR). We report on reliability (interobserver kappas of the 11 AMSTAR items), intraclass correlation coefficients (ICCs) of the sum scores, construct validity (ICCs of the sum scores of AMSTAR compared with those of other instruments), and completion times.

      Results

      The interrater agreement of the individual items of AMSTAR was substantial with a mean kappa of 0.70 (95% confidence interval [CI]: 0.57, 0.83) (range: 0.38–1.0). Kappas recorded for the other instruments were 0.63 (95% CI: 0.38, 0.78) for enhanced OQAQ and 0.40 (95% CI: 0.29, 0.50) for the Sacks' instrument. The ICC of the total score for AMSTAR was 0.84 (95% CI: 0.65, 0.92) compared with 0.91 (95% CI: 0.82, 0.96) for OQAQ and 0.86 (95% CI: 0.71, 0.94) for the Sacks' instrument. AMSTAR proved easy to apply, each review taking about 15 minutes to complete.

      Conclusions

      AMSTAR has good agreement, reliability, construct validity, and feasibility. These findings need confirmation by a broader range of assessors and a more diverse range of reviews.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Moher D.
        • Jadad A.R.
        • Nichol G.
        • Penman M.
        • Tugwell P.
        • Walsh S.
        Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists.
        Control Clin Trials. 1995; 16: 62-73
        • Shea B.
        • Dubé C.
        • Moher D.
        Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools.
        in: Egger M. Smith G.D. Altman D.G. Systematic reviews in health care: meta-analysis in context. BMJ Books, London2001: 122-139
        • Oxman A.D.
        • Guyatt G.H.
        Validation of an index of the quality of review articles.
        J Clin Epidemiol. 1991; 44: 1271-1278
        • Sacks H.
        • Berrier J.
        • Reitman D.
        • Ancona-Berk V.A.
        • Chalmers T.C.
        Meta-analyses of randomized controlled trials.
        N Engl J Med. 1987; 316: 450-455
        • Shea B.
        • Grimshaw J.
        • Wells G.
        • Boers M.
        • Andersson N.
        • Hamel C.
        • et al.
        Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
        BMC Med Res Methodol. 2007; 7: 10
        • Shea B.J.
        • Bouter L.M.
        • Peterson J.
        • Boers M.
        • Andersson N.
        • Ortiz Z.
        • et al.
        External validation of a measurement tool to assess systematic reviews (AMSTAR).
        PLoS ONE. 2007; 2 (Published online December 26, 2007): e1350https://doi.org/10.1371/journal.pone.0001350
        • Anonymous
        Effects of adjuvant tamoxifen and of cytotoxic therapy on mortality in early breast cancer. An overview of 61 randomized trials among 28,896 women. Early Breast Cancer Trialists' Collaborative Group.
        NEJM. 1989; 319: 1681-1692
        • Appel L.J.
        • Miller E.R.
        • Seidler A.J.
        • Whelton P.K.
        Does supplementation of diet with “fish oil” reduce blood pressure.
        Arch Intern Med. 1993; 153: 1429-1438
        • Buring J.E.
        • Evans D.A.
        • Mayrent S.L.
        • Rosner B.
        • Colton T.
        • Hennekens C.H.
        Randomized trials of aminoglycoside antibiotics: quantitative overview.
        Rev Inf Dis. 1988; 10: 951-957
        • Chalmers T.C.
        • Matta R.J.
        • Smith H.
        • Kunzler A.M.
        Evidence favouring the use of anticoagulants in the hospital phase of acute myocardial infarction.
        NEJM. 1977; 297: 1091-1096
        • Clagett G.P.
        • Reisch J.S.
        Prevention of venous thromboembolism in general surgical patients. Results of meta-analysis.
        Ann Surg. 1988; 208: 227-240
        • Counsell C.
        • Warlow C.
        • Naylor R.
        Different patches in carotoid surgery.
        Cochrane Library. 1996;
        • Daya S.
        Comparison of FSH and HMG in IVF.
        Cochrane Library. 1996;
        • Duley L.
        • Gulmezoglu A.M.
        • Henderson-Smart D.J.
        Anticonvulsants for pre-eclampsia.
        Cochrane Library. 1996;
        • Fanning J.
        • Bennett T.Z.
        • Hilgers R.D.
        Meta-analysis of cisplatin, doxorubicin, and cyclophosphamide versus cisplatin and cyclophosphamide chemotherapy of ovarian carcinoma.
        Obstet Gynecol. 1992; 80: 954-960
        • Gent M.
        • Roberts R.S.
        A meta-analysis of the studies of dihydroergotamine plus heparin in the prophylaxis of deep vein thrombosis.
        Chest. 1986; 89: 396S-400S
        • Gotzsche P.C.
        • Gjorup I.
        • Bonnen H.
        • Brahe N.E.
        • Becker U.
        • Burcharth F.
        Somatostatin vs placebo in bleeding oesophageal varices: randomised trial and meta-analysis.
        BMJ. 1995; 310: 1495-1498
        • Graves P.
        Malaria vaccines.
        Cochrane Library. 1996;
        • Henderson W.G.
        • Goldman S.
        • Copeland J.
        • Moritz T.E.
        • Harker L.A.
        Antiplatelet or anticoagulant therapy, after coronary artery bypass surgery: a meta-analysis of clinical trials.
        Ann Intern Med. 1989; 111: 743-750
        • Hodnett E.D.
        Alternative versus conventional delivery settings.
        Cochrane Library. 1996;
        • Hofmeyr G.J.
        Abdominal decompression.
        Cochrane Library. 1996;
        • Hopfenmuller W.
        Nackweis der therapeutischen Wirksamkeit eines Ginkgo biloba-Spezial extrakes: Meta-Analyse von 11 klinischen Studien bei Patienten mit Hirnleistungsstorungen im Alter.
        Arzneimittel-Forschung. 1994; 44: 1005-1013
        • Hughes E.
        • Fedorkow D.M.
        • Daya S.
        • Sagle M.A.
        • van de Kopple P.
        • Collins J.A.
        The routine use of gonadotropin-releasing hormone agonists prior to in vitro fertilization and gamete intra-fallopian transfer: a meta-analysis of randomized controlled trials.
        Fertil Steril. 1992; 58: 888-896
        • Kaufmann P.G.
        • Jacob R.G.
        • Ewart C.K.
        • Chesney M.A.
        • Muenz L.R.
        • Doub N.
        • et al.
        Hypertension Intervention Pooling Project.
        Health Psychol. 1988; 7: 209-224
        • Kramer M.S.
        Maternal antigen avoidance as lactation.
        Cochrane Library. 1996;
        • Lycka B.A.
        Postherpetic neuralgia and systemic corticosteroid therapy. Efficacy and safety.
        Int J Dermatol. 1990; 29: 523-527
        • McGrath J.J.
        • Soares K.V.S.
        Tardive dyskinesia and benzodiazepines.
        Cochrane Library. 1996;
        • Mulrow C.D.
        • Mulrow J.P.
        • Linn W.D.
        • Aguilar C.
        • Ramirez C.
        Relative efficacy of vasodilator therapy in chronic congestive heart failure. Implications of randomized trials.
        JAMA. 1988; 259: 3422-3426
        • Ohlsson A.
        Treatments of preterm premature rupture of the membranes: a meta-analysis.
        Am J Obstet Gynecol. 1989; 160: 890-906
        • Perez-Escamilla R.
        • Pollitt E.
        • Lonnerdal B.
        • Dewey K.G.
        Infant feeding policies in maternity wards and their effect on breast-feeding success: an analytical overview.
        Am J Public Health. 1994; 84: 89-97
        • Renfrew M.J.
        • Lang S.
        Breastfeeding and discharge times.
        Cochrane Library. 1996;
        • Renfrew M.J.
        • Lang S.
        Breastfeeding and early contact.
        Cochrane Library. 1996;
        • Soares K.V.S.
        • McGrath J.J.
        • Deeks J.J.
        Tardive dyskinesia and GABA agonist drugs.
        Cochrane Library. 1996;
        • Thacker S.B.
        Quality of controlled clinical trials. The case of imaging ultrasound in obstetrics: a review.
        BJOG. 1985; 92: 437-444
        • Velanovich V.
        Crystalloid versus colloid fluid resuscitation: a meta-analysis of mortality.
        Surgery. 1989; 105: 65-71
        • Wilson A.P.R.
        • Shrimpton S.
        • Jaderberg M.
        A meta-analysis of the use of amoxycillin-clavulanic acid in surgical prophylaxis.
        J Hosp Infect. 1992; 22: 9-21
        • Cohen J.
        A coefficient of agreement for nominal scales.
        Educ Psychol Meas. 1960; 20: 37-46
        • Bland J.M.
        • Altman D.G.
        Statistical methods for assessing agreement between two methods of clinical measurement.
        Lancet. 1986; i: 307-310
        • Bland J.M.
        • Altman D.G.
        Statistical methods for assessing agreement between measurements.
        Biochim Clin. 1987; 11: 399-404
      1. Anonymous. This week's citation classic: comparing methods of clinical measurement. Curr Contents 1992; CC/NUMBER 40: 8. Available at http://garfield.library.upenn.edu/classics1992/A1992JN24800001.pdf.

        • Uebersax J.S.
        Diversity of decision-making models and the measurement of inter-rater agreement.
        Psychol Bull. 1987; 101: 140-146
        • Cohen J.
        Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit.
        Psychol Bull. 1968; 70: 213-220
        • Tugwell P.
        • Bombardier C.
        A methodological framework for developing and selecting endpoints in clinical trials.
        J Rheumatol. 1982; 9: 758-762
        • Singh S.
        • Bai A.
        • Lal A.
        • Yu C.
        • Ahmed F.
        Developing evidence-based best practices for the prescribing and use of proton pump inhibitors in Canada.
        The Canadian Agency for Drugs and Technologies in Health (CADTH), Ottawa, Canada2006
        • Balk E.
        • Bonis P.
        • Moskowitz H.
        • Schmid C.
        • Ioannidis J.
        • Wang C.
        • et al.
        Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials.
        JAMA. 2002; 287: 2973-2982
        • Jüni P.
        • Altman D.G.
        • Egger M.
        Assessing the quality of controlled clinical trials.
        in: Egger M. Davey Smith G. Altman D.G. Systematic reviews in health care: meta-analysis in context. 2nd ed. BMJ Books, London2001
        • Barnes D.E.
        • Bero L.A.
        Why review articles on the health effects of passive smoking reach different conclusions.
        JAMA. 1998; 279: 1566-1570
        • Biondi-Zoccai G.
        • Lotrionte M.
        • Abbate A.
        • Testa L.
        Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study.
        BMJ. 2006; 332: 202-206
        • Chou R.
        • Helfand M.
        Challenges in systematic reviews that assess treatment harms.
        Ann Intern Med. 2005; 142: 1090-1099
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective Publication of antidepressant trials and its influence on apparent efficacy.
        N Eng J Med. 2008; 358: 252-260
        • Whittington C.J.
        • Kendall T.
        • Fonagy P.
        • Cottrell D.
        • Cotgrove A.
        • Boddington E.
        Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data.
        Lancet. 2004; 363: 1341-1345
        • McGinn T.
        • Guyatt G.
        • Cook R.
        • Meade M.
        Diagnosis: measuring agreement beyond chance.
        in: Guyatt G. Rennie D. Users' guide to the medical literature. A manual for evidence-based clinical practice. AMA Press, Chicago, IL2002: 461-470
        • Moher D.
        • Cook D.J.
        • Eastwood S.
        • Olkin I.
        • Rennie D.
        Stroup D for the QUOROM group. Improving the reporting quality of meta-analysis of randomized controlled trials: the QUOROM statement.
        Lancet. 1999; 354: 1896-1900
        • Oxman A.D.
        • Schünemann H.J.
        • Fretheim A.
        Improving the use of research evidence in guideline development: synthesis and presentation of evidence.
        Health Res Policy Syst. 2006; (Received April 7, 2006, Accepted December 5, 2006. Available at): 20https://doi.org/10.1186/1478-4505-4-20