Validation of diagnostic codes within medical services claims

  • Machelle Wilchesky
    Department of Epidemiology and Biostatistics, McGill University, Morrice House, 1140 Pine Avenue West, Montreal, QCH3A 1A3, Canada
    Search for articles by this author
  • Robyn M Tamblyn
    Corresponding author. Tel.: 514-843-2831; fax: 514-843-1551.
    Department of Epidemiology and Biostatistics, McGill University, Morrice House, 1140 Pine Avenue West, Montreal, QCH3A 1A3, Canada

    Department of Medicine, McGill University, McGill University Health Center, 687 Pine Avenue West, Montreal, QCH3A 1A1, Canada
    Search for articles by this author
  • Allen Huang
    Division of Geriatrics, Department of Medicine, McGill University, McGill University Health Center, 687 Pine Avenue West, Montreal, QCH3A 1A3, Canada
    Search for articles by this author



      Few studies have attempted to validate the diagnostic information contained within medical service claims data, and only a small proportion of these have attempted to do so using the medical chart as a gold standard. The goal of this study is to determine the sensitivity and specificity of medical services claims diagnoses for surveillance of 14 drug disease contraindications used in drug utilization review, the Charlson comorbidity index and the Johns Hopkins Adjusted Care Group Case-Mix profile (ADGs).

      Study design and setting

      Diagnoses were abstracted from the medical charts of 14,980 patients, and were used as the “gold standard,” against which diagnoses obtained from the administrative database for the same patients were compared.


      Conditions associated with drug disease contraindications with the exception of hypertension and chronic obstructive pulmonary disease (COPD) showed a specificity of 90% or higher. Sensitivity of claims data was substantially lower, with glaucoma, hypertension, and diabetes being the most sensitive conditions at 76, 69, and 64%, respectively. Each of the 18 disease conditions contained in the Charlson comorbidity index showed high specificity, but sensitivity was more variable among conditions as well as by coding definitions. Although ADG specificity was also high, the vast majority of ADGs had sensitivities of less than 60%.


      The administrative data was found to have diagnoses and conditions that were highly specific but that vary greatly by condition in terms of sensitivity. To appropriately obtain diagnostic profiles, it is recommended that data pertaining to all physician billings be used.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Roos N.P
        • Roos L.L
        • Mossey J
        • Havens B
        Using administrative data to predict important health outcomes: entry to hospital, nursing home, and death.
        Med Care. 1988; 26: 221-239
        • Coleman A.L
        • Morgenstern H
        Use of insurance claims databases to evaluate the outcomes of ophthalmic surgery.
        Surv Ophthalmol. 1997; l42: 271-278
        • Leader S.G
        • Mallick R
        • Briggs N.C
        Myocardial infarction in newly diagnosed hypertensive Medicaid patients free of coronary heart disease and treated with calcium channel blockers.
        Am J Med. 1997; 102: 150-157
        • Mitchell J.B
        • Ballard D.J
        • Whisnant J.P
        • Ammering C.J
        • Matchar D.B
        • Samsa G.P
        Using physician claims to identify postoperative complications of carotid endarterectomy.
        Health Serv Res. 1996; 31: 141-152
        • Young W.W
        • Marks S.M
        • Kohler S.A
        • Hsu A.Y
        Dissemination of clinical results. Mastectomy versus lumpectomy and radiation therapy.
        Med Care. 1996; 34: 1003-1017
        • Melfi C
        • Holleman E
        • Arthur D
        • Katz B
        Selecting s patient characteristics index for the prediction of medical outcomes using administrative claims data.
        J Clin Epidemiol. 1995; 48: 917-926
        • White L.L
        • Holimon T.D
        • Tepedino J.T
        • Portner T.S
        • Wan J.Y
        • Thompson J.W
        Antimicrobials prescribed for otitis media in a pediatric Medicaid population.
        Am J Health Syst Pharm. 1995; 53: 2963-2969
        • Weiner J
        • Starfield B
        • Powe N.R
        • Stuart M
        • Steinwachs D
        Ambulatory care practice variation within a medicaid program.
        Health Serv Res. 1996; 30: 751-770
      1. Herz EJ, Chawla AJ, Sredl K. Changes in the receipt of age-appropriate immunizations among young children under Medicaid 1989 and 1992. International Conference on Health Policy Research. Association of Health Services Research Conference; December 5–7, 1997.

        • Rosenbach M.L
        • Hermann R.C
        • Dorwart R.A
        Use of electroconvulsive therapy in the Medicare population between 1987 and 1992.
        Psychiatr Serv. 1997; 48: 1537-1542
        • Welch W.P
        • Verrilli D
        • Katz S.J
        • Latimer E
        A detailed comparison of physician services for the elderly in the United States and Canada.
        JAMA. 1996; 275: 1410-1416
        • Ives D.G
        • Lave J.R
        • Traven N.D
        • Schulz R
        • Kuller L.H
        Mammography and pap smear use by older rural women.
        Public Health Rep. 1996; 111: 244-250
        • Grootendorst P
        Health care policy evaluation using longitudinal insurance claims data: an application of the panel ToBit Estimator.
        Health Econ. 1997; 6: 365-382
        • Gavin N.I
        • Adams E.K
        • Herz E.J
        • Chawla A.J
        • Ellwood M.R
        • Hill I.T
        • Zimmerman B.L
        • Wasserman J
        The use of EPSDT and other health care services by children enrolled in Medicaid: the impact of OBRA'89.
        Milbank Q. 1998; 76: 207-250
        • Maskarinec G
        Diabetes in Hawaii: estimating prevalence from insurance claims data.
        Am J Public Health. 1997; 87: 1717-1720
        • Manfreda J
        • Becker A.B
        • Wang P.Z
        • Roos L.L
        • Anthonisen N.R
        Trends in physician-diagnosed asthma prevalence in Manitoba between 1980 and 1990.
        Chest. 1993; 103: 151-157
        • Lestina D.C
        • Miller T.R
        • Smith G.S
        Creating injury episodes using medical claims data.
        J Trauma. 1998; 45: 565-569
        • Dighe M.S
        • Bennett L.T
        • Rappaport H.M
        Economic evaluation of Medi-Cal epilepsy patients on monotherapy or polytherapy in 1994–95.
        AHSR.FHSR Annu Meet Abstr Book. 1996; 13
        • Xuan J
        • Duong P.T
        • Russo P.A
        • Lacey M.J
        • Wong B
        The economic burden of congestive heart failure in a managed care population.
        Am J Managed Care. 2000; 6: 693-700
        • Weiner J.P
        • Dobson A
        • Maxwell S.L
        • Coleman K
        • Starfield B
        • Anderson G.F
        Risk-adjusted Medicare capitation rates using ambulatory and inpatient diagnoses.
        Health Care Financ Rev. 1996; 17: 77-99
        • Garnick D.W
        • Fowles J
        • Lawthers A.G
        • Weiner J.P
        • Parente S.T
        • Palmer R.H
        Focus on quality: profiling physicians' practice patterns.
        J Ambul Care Manage. 1994; 17: 44-75
        • Parente S.T
        • Weiner J.P
        • Garnick D.W
        • Richards T.M
        • Fowles J
        • Lawthers A.G
        • Chandler P
        • Palmer R.H
        Developing a quality improvement database using health insurance data: a guided tour with application to Medicare's National Claims History file.
        Am J Med Qual. 1995; 10: 162-176
        • Weiner J.P
        • Parente S.T
        • Garnick D.W
        • Fowles J
        • Lawthers A.G
        • Palmer H.R
        Variation in office-based quality: a claims-based profile of care provided to Medicare patients with diabetes.
        JAMA. 1995; 273: 1503-1508
        • Parente S.T
        • Weiner J
        • Garnick D.W
        • Fowles J
        • Lawthers A.G
        • Palmer R.H
        Profiling resource use by primary-care practices: managed medical implications.
        Health Care Financ Rev. 1996; 17: 23-42
        • Ozminkowski R.J
        • Noether M
        • Nathanson P
        • Smith K.M
        • Raney B.E
        • Mickey D
        • Hawley Jr., P.M
        Profiling primary care physicians for a new managed care network.
        Health Serv Manage Res. 1997; 10: 173-186
        • Powe N.R
        • Weiner J.P
        • Starfield B
        • Stuart M
        • Baker A
        • Steinwachs D.M
        Systemwide provider performance in a Medicaid program. Profiling the care of patients with chronic illnesses.
        Med Care. 1996; 34: 798-810
        • Starfield B
        • Powe N.R
        • Weiner J.R
        • Stuart M
        • Steinwachs D
        • Hudson-Scholle S
        • Gerstenberger A
        Costs vs quality in different types of primary care settings.
        JAMA. 1994; 272: 1903-1908
        • Dhanani N
        • Litwin M.S
        The effect of physician specialty on resource use and outcomes.
        AHSR.FHSR Annu Meet Abstr Book. 1996; 13: 4a-5a
        • Chang W
        • McCracken S.B
        Apply case mix adjustment in physician performance profiling using large administrative database.
        AHSR. FHSR Annu Meet Abstr Book. 1996; 13: 160-161
        • Muhajarine N
        • Mustard C
        • Roos L.L
        • Young K
        • Gelskey D.E
        Comparison of survey and physician claims data for detecting hypertension.
        J Clin Epidemiol. 1997; 50: 711-718
        • Roos L.L
        • Roos N.P
        • Cageorge S.M
        • Nicol P
        How good are the data? Reliability of one health care data bank.
        Med Care. 1982; 20: 266-276
        • Klabunde C.N
        • Potosky A.L
        • Legler J.M
        • Warren J.L
        Development of a comorbidity index using physician claims data.
        J Clin Epidemiol. 2000; 53: 1258-1267
        • Schneeweiss S
        • Maclure M
        Use of comorbidity scores for control of confounding in studies using administrative databases.
        Int J Epidemiol. 2000; 29: 891-898
        • Schneeweiss S
        • Seeger J.D
        • Maclure M
        • Wang P.S
        • Avorn J
        • Glynn R.J
        Performance of comorbidity scores to control for confounding in epidemiologic studies using claims data.
        Am J Epidemiol. 2001; 154: 854-864
        • Klabunde C.N
        • Warren J.L
        • Legler J.M
        Assessing comorbidity using claims data.
        Med Care. 2002; 40: IV-26-IV-35
        • Stukenborg G.J
        • Wagner D.P
        • Connors Jr., A.F
        Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations.
        Med Care. 2001; 39: 727-739
        • Reid R.J
        • MacWilliam L
        • Verhulst L
        • Roos N
        • Atkinson M
        Performance of the ACG case-mix system in two Canadian provinces.
        Med Care. 2001; 39: 86-99
        • Rosen A.K
        • Loveland S
        • Anderson J.J
        • Rothendler J.A
        • Hankin C.S
        • Rakovski C.C
        • Moskowitz M.A
        • Berlowitz D.R
        Evaluating diagnosis-based case-mix measures: how well do they apply to the VA population?.
        Med Care. 2001; 39: 692-704
        • Majeed A
        • Bindman A.B
        • Weiner J.P
        Use of risk adjustment in setting budgets and measuring performance in primary care I: how it works.
        Br Med J. 2001; 323: 604-607
        • Sorbero M.E
        • Dick A.W
        • Zwanziger J
        • Mukamel D
        • Weyl N
        The effect of capitation on switching primary care physicians.
        Health Serv Res. 2003; 38: 191-209
        • Kralewski J.E
        • Rich E.C
        • Feldman R
        • Dowd B.E
        • Bernhardt T
        • Johnson C
        • Gold W
        The effects of medical group practice and physician payment methods on costs of care.
        Health Serv Res. 2000; 35: 591-613
        • van Doorn C
        • Bogardus S.T
        • Williams C.S
        • Concato J
        • Towle V.R
        • Inouye S.K
        Risk adjustment for older hospitalized persons: a comparison of two methods of data collection for the Charlson index.
        J Clin Epidemiol. 2001; 54: 694-701
        • Robinson J.R
        • Young T.K
        • Roos L.L
        • Gelskey D.E
        Estimating the burden of disease. Comparing administrative data and self-reports.
        Med Care. 1997; 35: 932-947
        • Colditz G.A
        • Martin P
        • Stampfer M.J
        • Willett W.C
        • Sampson L
        • Rosner B
        • Hennekens C.H
        • Speizer F.E
        Validation of questionnaire information on risk factors and disease outcomes in a prospective cohort study of women.
        Am J Epidemiol. 1986; 123: 894-900
        • Katz J.N
        • Barrett J
        • Liang M.H
        • Bacon A.M
        • Kaplan H
        • Kieval R.I
        • Lindsey S.M
        • Roberts W.N
        • Sheff D.M
        • Spencer R.T
        • Weaver A.L
        • Baron J.A
        Sensitivity and positive predictive value of Medicare Part B physician claims for rheumatologic diagnoses and procedures.
        Arthritis Rheum. 1997; 40: 1594-1600
        • Worth R.M
        • Mytinger R.E
        Medical insurance claims as a source of data for research: accuracy of diagnostic coding.
        Hawaii Med J. 1996; 55: 9-11
        • Cooper G.S
        • Yuan Z
        • Stange K.C
        • Dennis L.K
        • Amini S.B
        • Rimm A.A
        The sensitivity of Medicare claims data for case ascertainment of six common cancers.
        Med Care. 1999; 37: 436-444
        • Charlson M.E
        • Pompei P
        • Ales K.L
        • MacKenzie C.R
        A new method of classifying prognostic comorbidity in longitudinal studies: development and validation.
        J Chronic Dis. 1987; 40: 373-383
        • Starfield B
        • Weiner J
        • Mumford L
        • Steinwachs D
        Ambulatory care groups: a categorization of diagnoses for research and management.
        Health Serv Res. 1991; 26: 53-74
        • Tamblyn R.M
        • Jacques A
        • Laprise R
        • Huang A
        • Perreault R
        The Office of the Future Project: the integration of new technology into office practice—academic detailing through the super highway.
        Clin Perform Qual Health Care. 1997; 5: 104-108
        • Majeed A
        • Bindman A.B
        • Weiner J.P
        Use of risk adjustment in setting budgets and measuring performance in primary care II: advantages, disadvantages, and practicalities.
        Br Med J. 2001; 323: 607-610
        • Lipton H.L
        • Bird J.A
        Drug utilization review in ambulatory settings: state of the science and directions for outcomes research.
        Med Care. 1993; 12: 1069-1082
        • Ashton H
        Psychotropic-drug prescribing for women.
        Br J Psychiatry. 1991; 158: 30-35
        • Smith N.S
        • Weiner J.P
        Applying population-based case mix adjustment in managed care: the Johns Hopkins Ambulatory Care Group system.
        Manag Care Q. 1994; 2: 21-34
        • Romanow R.J
        Building on values: the future of health care in Canada—final report (Romanow). Commission on the Future of Health Care in Canada.
        Health Care Commission, Saskatoon, Saskatchewan2002 (p. 1–339)
        • Régie de l'assurance-maladie
        Statistiques Annuelles 1996.
        Government of Quebec, Quebec1996
        • Tamblyn R
        • Huang A
        • Perreault R
        • Jacques A
        • Roy D
        • Hanley J
        • McLeod P
        • Laprise R
        The medical office of the twenty-first century (MOXXI): evaluation of the effectiveness of computerised decision support in reducing inappropriate prescribing in primary care.
        Can Med Assoc J. 2003; (169:549–56)
        • USAGE (Groupe de recherche du Québec sur l'utilization des médicaments chez les personnes âgées)
        Réduction des ordonnances potentiellement non appropriées chez les personnes âgées (ROPNAPA)—Medical Office of the 21st Century (MOXXI).
        McGill University, Quebec, Canada1998
        • McLeod P
        • Huang A
        • Tamblyn R
        • Gayton D
        Defining inappropriate practices in prescribing for elderly people: a national consensus panel.
        CMA. 1997; 156: 385-391
        • Deyo R.A
        • Cherkin D.C
        • Ciol M.A
        Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.
        J Clin Epidemiol. 1992; 45: 613-619
        • Romano P.S
        • Roos L.L
        • Jollis J.G
        Adapting a clinical comorbidity index for use with ICD-9-CM administrative data. Differing perspectives.
        J Clin Epidemiol. 1993; 46: 1075-1079
        • Cleves M.A
        • Sanchez N
        • Draheim M
        Evaluation of two competing methods for calculating Charlson's comorbidity index when analyzing short-term mortality using administrative data.
        J Clin Epidemiol. 1997; 50: 903-908
        • Last J.M
        A dictionary of epidemiology.
        3rd ed. Oxford University Press, New York1983
        • Ramsay J.O
        • Abrahamowicz M
        Binomial regression with monotone splines: a psychometric application.
        J Am Stat Assoc. 1989; 84: 906-915
        • Vollset S.E
        Confidence intervals for a binomial proportion.
        Stat Med. 1993; 12: 809-824
        • Ghosh B.K
        A comparison of some approximate confidence intervals for the binomial parameter.
        J Am Stat Assoc. 1979; 74: 894-900
        • Edwardes M.D
        Confidence intervals for a binomial proportion: reader reaction (letter; comment).
        Stat Med. 1994; 13: 1693-1698
        • Santer T.J
        • Duffy D.E
        The statistical analysis of discrete data.
        Springer-Verlag, New York1989
        • Edwardes M.D
        The evaluation of confidence sets with application to binomial intervals.
        Stat Sinica. 1998; 8: 393-409
        • RAMQ
        Tableau 2.13: nombre de participants, nombre de services médicaux et leur coût par participant selon le sexe, le groupe d'âge et le type de service, rémunération à l'acte, médecine et chirurgie, Québec 2000.
        Services médicaux, Quebec2000
        • Rethans J.J
        • Martin E
        • Metsemakers J
        To what extent do clinical notes by general practioners reflect actual medical performance? A study using simulated patients.
        Br J Gen Pract. 1994; 44: 153-156
        • Norman G.R
        • Neufeld V.R
        • Walsh A
        • Woodward C.A
        • McConvey G.A
        Measuring physicians' performances by using simulated patients.
        J Med Educ. 1985; 60: 925-934