Measuring quality of care: considering conceptual approaches to quality indicator development and evaluation

  • Henry T. Stelfox
    Corresponding author. Tel.: +403-944-2334; fax: +403-283-9994.
    Department of Critical Care Medicine, Institute for Public Health, University of Calgary, Teaching Research & Wellness Building, 3280 Hospital Drive NW, Calgary, Alberta, Canada T2N 4Z6

    Department of Medicine, Institute for Public Health, University of Calgary, Teaching Research & Wellness Building, 3280 Hospital Drive NW, Calgary, Alberta, Canada T2N 4Z6

    Department of Community Health Sciences, Institute for Public Health, University of Calgary, Teaching Research & Wellness Building, 3280 Hospital Drive NW, Calgary, Alberta, Canada T2N 4Z6
    Search for articles by this author
  • Sharon E. Straus
    Department of Medicine, Li Ka Shing Knowledge Institute of Saint Michael's Hospital, University of Toronto, Toronto, Ontario, Canada
    Search for articles by this author
Published:September 09, 2013DOI:



      In this article, we describe one approach for developing and evaluating quality indicators.

      Study Design and Setting

      We focus on describing different conceptual approaches to quality indicator development, review one approach for developing quality indicators, outline how to evaluate quality indicators once developed, and discuss quality indicator maintenance.


      The key steps for developing quality indicators include specifying a clear goal for the indicators; using methodologies to incorporate evidence, expertise, and patient perspectives; and considering contextual factors and logistics of implementation. The Strategic Framework Board and the National Quality Measure Clearinghouse have developed criteria for evaluating quality indicators that complement traditional psychometric evaluations. Optimal strategies for quality indicator maintenance and dissemination have not been determined, but experiences with clinical guideline maintenance may be informative.


      For quality indicators to effectively guide quality improvement efforts, they must be developed, evaluated, maintained, and implemented using rigorous evidence-informed practices.


      To read this article in full you will need to make a payment


        • Canadian Institute for Health Information
        Pan-Canadian primary health care indicators.
        Canadian Institute for Health Information, Ottawa, Ontario2006
        • Canadian Association for Health Policy and Services Research
        Updating Pan-Canadian primary health care indicators—what we've learned.
        Canadian Institute for Health Information, Ottawa, Ontario2012
      1. Agency for Healthcare Research and Quality. AHRQ quality indicators. 2012; Available at Accessed April 25, 2013.

        • Krumholz H.M.
        • Anderson J.L.
        • Brooks N.H.
        • Fesmire F.M.
        • Lambrew C.T.
        • Landrum M.B.
        • et al.
        ACC/AHA clinical performance measures for adults with ST-elevation and non-ST-elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures on ST-Elevation and Non-ST-Elevation Myocardial Infarction).
        Circulation. 2006; 113: 732-761
      2. Centers for Medicare and Medicaid Services Hospitals Compare. Centers for Medicare and Medicaid Services; 2012; Available at Accessed August 3, 2012.

        • Jollis J.G.
        • Roettig M.L.
        • Aluko A.O.
        • Anstrom K.J.
        • Applegate R.J.
        • Babb J.D.
        • et al.
        Implementation of a statewide system for coronary reperfusion for ST-segment elevation myocardial infarction.
        JAMA. 2007; 298: 2371-2380
        • Morris C.G.
        • McCoy E.
        Clearing the cervical spine in unconscious polytrauma victims, balancing risks and effective screening.
        Anaesthesia. 2004; 59: 464-482
      3. Santana MJ, Stelfox HT. Development and evaluation of evidence-informed indicators for adult injury care. Ann Surg. 2013. Epub ahead of print.

        • Stelfox H.T.
        • Velmahos G.C.
        • Gettings E.
        • Bigatello L.M.
        • Schmidt U.
        Computed tomography for early and safe discontinuation of cervical spine immobilization in obtunded multiply injured patients.
        J Trauma. 2007; 63: 630-636
        • Stelfox H.T.
        • Straus S.E.
        • Nathens A.
        • Gruen R.L.
        • Hameed S.M.
        • Kirkpatrick A.
        Trauma center quality improvement programs in the United States, Canada, and Australasia.
        Ann Surg. 2012; 256: 163-169
        • Pasic R.P.
        • Kantardzie M.
        • Templeman C.
        • Levine R.L.
        Insufflation techniques in gynecologic laparoscopy.
        Surg Laparosc Endosc Percutan Tech. 2006; 16: 18-23
        • Osler M.
        Kliniske databaser i social epidemiologisk forskning.
        Ugeskr Laeger. 2009; 171: 1843-1846
        • Jones A.R.
        Data mining can support quality assurance.
        J R Soc Med. 2009; 102: 358-359
        • Putnam W.
        • Bower K.N.
        • Cox J.
        • Twohig P.
        • Pottie K.
        • Jackson L.
        • et al.
        Quality indicators for cardiac care: national standards in a community context.
        J Health Serv Res Policy. 2006; 11: 5-12
        • Davies S.M.
        • Geppert J.
        • McClellan M.
        • McDonald K.M.
        • Romano P.S.
        • Shojania K.G.
        Refinement of the HCUP quality indicators.
        (Rockville, MD; Agency for Healthcare Research and Quality;)2001
        • Jones J.
        • Hunter D.
        Consensus methods for medical and health services research.
        BMJ. 1995; 311: 376-380
        • Black N.
        • Murphy M.
        • Lamping D.
        • McKee M.
        • Sanderson C.
        • Askham J.
        • et al.
        Consensus development methods: a review of best practice in creating clinical guidelines.
        J Health Serv Res Policy. 1999; 4: 236-248
        • Kahan J.P.
        • Park R.E.
        • Leape L.L.
        • Bernstein S.J.
        • Hilborne L.H.
        • Parker L.
        • et al.
        Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures.
        Med Care. 1996; 34: 512-523
        • Coulter I.
        • Adams A.
        • Shekelle P.
        Impact of varying panel membership on ratings of appropriateness in consensus panels: a comparison of a multi- and single disciplinary panel.
        Health Serv Res. 1995; 30: 577-591
        • Leape L.L.
        • Park R.E.
        • Kahan J.P.
        • Brook R.H.
        Group judgments of appropriateness: the effect of panel composition.
        Qual Assur Health Care. 1992; 4: 151-159
        • Campbell S.M.
        • Hann M.
        • Roland M.O.
        • Quayle J.A.
        • Shekelle P.G.
        The effect of panel membership and feedback on ratings in a two-round Delphi survey: results of a randomized controlled trial.
        Med Care. 1999; 37: 964-968
        • Nilsen E.S.
        • Myrhaug H.T.
        • Johansen M.
        • Oliver S.
        • Oxman A.D.
        Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material.
        Cochrane Database Syst Rev. 2006; 3: CD004563
        • Boivin A.
        • Lehoux P.
        • Lacombe R.
        • Lacasse A.
        • Burgers J.
        • Grol R.
        Target for improvement: a cluster randomised trial of public involvement in quality-indicator prioritisation (intervention development and study protocol).
        Implement Sci. 2011; 6: 45
        • McGlynn E.A.
        Selecting common measures of quality and system performance.
        Med Care. 2003; 41: I39-I47
        • The Joint Commission
        Attributes of core performance measures and associated evaluation criteria.
        The Joint Commission, 2008 (Available at) (Accessed August 3, 2012)
        • National Quality Measures Clearinghouse
        Domain framework and inclusion criteria.
        Agency for Healthcare Research and quality, Rockville, MD2012 (Available at) (Accessed August 3, 2012)
        • Kotter T.
        • Blozik E.
        • Scherer M.
        Methods for the guideline-based development of quality indicators—a systematic review.
        Implement Sci. 2012; 7: 21
        • Farquhar M.
        AHRQ quality indicators.
        in: Hughes R. Patient safety and quality: an evidence-based handbook for nurses. Agency for Healthcare Research and Quality, Rockville, MD2008: 341-367
        • Campbell S.M.
        • Braspenning J.
        • Hutchinson A.
        • Marshall M.
        Research methods used in developing and applying quality indicators in primary care.
        Qual Saf Health Care. 2002; 11: 358-364
        • McDowell I.
        Measuring health: a guide to rating scales and questionnaires.
        3rd ed. Oxford University Press, New York2006
        • Quan H.
        • Sundararajan V.
        • Halfon P.
        • Fong A.
        • Burnand B.
        • Luthi J.C.
        • et al.
        Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.
        Med Care. 2005; 43: 1130-1139
        • Bobrovitz N.
        • Parrilla J.S.
        • Santana M.J.
        • Straus S.E.
        • Stelfox H.T.
        A qualitative analysis of a consensus process to develop quality indicators of injury care.
        Implement Sci. 2013; 8: 45
        • National Quality Measures Clearinghouse
        Validity of clinical quality measures.
        Agency for Healthcare Research and quality, Rockville, MD2012 (Available at) (Accessed August 3, 2012)
        • McGlynn E.A.
        • Asch S.M.
        Developing a clinical performance measure.
        Am J Prev Med. 1998; 14: 14-21
        • Streiner D.L.
        • Norma G.R.
        Health measurement scales: a practical guide to their development and use.
        Oxford University Press, Oxford, UK1989
        • Huff E.D.
        Comprehensive reliability assessment and comparison of quality indicators and their components.
        J Clin Epidemiol. 1997; 50: 1395-1404
        • Scinto J.D.
        • Galusha D.H.
        • Krumholz H.M.
        • Meehan T.P.
        The case for comprehensive quality indicator reliability assessment.
        J Clin Epidemiol. 2001; 54: 1103-1111
        • Bartko J.J.
        The intraclass correlation coefficient as a measure of reliability.
        Psychol Rep. 1966; 19: 3-11
        • Deyo R.A.
        • Diehr P.
        • Patrick D.L.
        Reproducibility and responsiveness of health status measures. Statistics and strategies for evaluation.
        Control Clin Trials. 1991; 12: 142S-158S
        • McNeil B.J.
        • Keller E.
        • Adelstein S.J.
        Primer on certain elements of medical decision making.
        N Engl J Med. 1975; 293: 211-215
        • Iron K.
        • Manuel D.G.
        Quality assessment of administrative data (QuAAD): an opportunity for enhancing Ontario's health data.
        Institute for Clinical Evaluative Sciences, Toronto, Canada2007
        • Juurlink D.
        • Preyra C.
        • Croxford R.
        • Chong A.
        • Austin P.
        • Tu J.
        • et al.
        Canadian institute for health information discharge abstract database: a validation study.
        Institute for Clinical Evaluative Sciences, Toronto, Canada2006
        • Murphy J.M.
        • Berwick D.M.
        • Weinstein M.C.
        • Borus J.F.
        • Budman S.H.
        • Klerman G.L.
        Performance of screening and diagnostic tests. Application of receiver operating characteristic analysis.
        Arch Gen Psychiatry. 1987; 44: 550-555
        • Swets J.A.
        Measuring the accuracy of diagnostic systems.
        Science. 1988; 240: 1285-1293
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
      4. Iezzoni L.I. Risk adjustment for measuring health care outcomes. 3rd ed. Health Administration Press, Chicago, IL2003
        • Tu J.V.
        • Donovan L.R.
        • Lee D.S.
        • Wang J.T.
        • Austin P.C.
        • Alter D.A.
        • et al.
        Effectiveness of public report cards for improving the quality of cardiac care: the EFFECT study: a randomized trial.
        JAMA. 2009; 302: 2330-2337
        • Hibbard J.H.
        • Stockard J.
        • Tusler M.
        Hospital performance reports: Impact on quality, market share, and reputation.
        Health Aff (Millwood). 2005; 24: 1150-1160
        • National Guideline Clearinghouse
        Inclusion criteria.
        Agency for Healthcare Research and Quality, Rockville, MD2012 ( Accessed August 3, 2012.)
        • Shekelle P.G.
        • Ortiz E.
        • Rhodes S.
        • Morton S.C.
        • Eccles M.P.
        • Grimshaw J.M.
        • et al.
        Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated?.
        JAMA. 2001; 286: 1461-1467
        • Shojania K.G.
        • Sampson M.
        • Ansari M.T.
        • Ji J.
        • Garritty C.
        • Rader T.
        • et al.
        Updating systematic reviews.
        (Rockville, MD; Agency for Healthcare Research and Quality;)2007
        • Mattke S.
        When should measures be updated? Development of a conceptual framework for maintenance of quality-of-care measures.
        Qual Saf Health Care. 2008; 17: 182-186
        • Shekelle P.
        • Eccles M.P.
        • Grimshaw J.M.
        • Woolf S.H.
        When should clinical guidelines be updated?.
        BMJ. 2001; 323: 155-157
        • Gartlehner G.
        • West S.L.
        • Lohr K.N.
        • Kahwati L.
        • Johnson J.G.
        • Harris R.P.
        • et al.
        Assessing the need to update prevention guidelines: a comparison of two methods.
        Int J Qual Health Care. 2004; 16: 399-406
        • Straus S.E.
        • Tetroe J.
        • Graham I.A.
        Introduction to knowledge translation.
        in: Straus S.E. Tetroe J. Graham I.A. Knowledge translation in health care: moving from evidence to practice. Wiley/Blackwell/BMJ, Oxford, UK2009
      5. American College of Cardiology. Guidelines & quality standard. 2012; Available at Accessed August 3, 2012.

        • Fitch K.
        • Bernstein S.J.
        • Aguilar M.D.
        • Burnand B.
        • LaCalle J.R.
        • Lazaro P.
        • et al.
        The RAND/UCLA appropriateness method user's manual.
        RAND, Santa Monica, CA2001

      Linked Article