If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Division of Rheumatology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, CanadaMilan Ilich Arthritis Research Centre, 5591 No. 3 Rd, Richmond, British Columbia, Canada V6X 2C7
Division of Rheumatology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, CanadaMilan Ilich Arthritis Research Centre, 5591 No. 3 Rd, Richmond, British Columbia, Canada V6X 2C7
Division of Rheumatology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, CanadaMilan Ilich Arthritis Research Centre, 5591 No. 3 Rd, Richmond, British Columbia, Canada V6X 2C7
To conduct a systematic review of studies reporting on the development or validation of comorbidity indices using administrative health data and compare their ability to predict outcomes related to comorbidity (ie, construct validity).
Study Design and Setting
We conducted a comprehensive literature search of MEDLINE and EMBASE, until September 2012. After title and abstract screen, relevant articles were selected for review by two independent investigators. Predictive validity and model fit were measured using c-statistic for dichotomous outcomes and R2 for continuous outcomes.
Results
Our review includes 76 articles. Two categories of comorbidity indices were identified: those identifying comorbidities based on diagnoses, using International Classification of Disease codes from hospitalization or outpatient data, and based on medications, using pharmacy data. The ability of indices studied to predict morbidity-related outcomes ranged from poor (C statistic ≤0.69) to excellent (C statistic >0.80) depending on the specific index, outcome measured, and study population. Diagnosis-based measures, particularly the Elixhauser Index and the Romano adaptation of the Charlson Index, resulted in higher ability to predict mortality outcomes. Medication-based indices, such as the Chronic Disease Score, demonstrated better performance for predicting health care utilization.
Conclusion
A number of valid comorbidity indices derived from administrative data are available. Selection of an appropriate index should take into account the type of data available, study population, and specific outcome of interest.
A number of comorbidity indices are available for use in studies with administrative health data, in order to control for the overall burden of comorbidities.
•
To guide researchers and health policy makers in selecting the index most appropriate for their purpose, this systematic review describes the conceptual and methodological differences among the various indices and compares their ability to predict outcomes related to comorbidity (i.e. construct validity).
•
The review reveals that a number of comorbidity indices demonstrate validity in predicting mortality.
•
A diagnosis-based index, such as the Quan- or van Walraven- EI or Romano-CCI, is recommended in studies where the outcome of interest is mortality.
•
For studies evaluating healthcare utilization, where medication data is available, a medication-based index, such as the RxRisk-V, is recommended.
Administrative databases are being increasingly used for research purposes. They play an important role in epidemiologic, quality of care, pharmacovigilance, and outcome studies. These databases provide complementary information to randomized controlled trials because of their real-life setting, large samples, long follow-up duration, and their ability to provide population-based samples, free of selection bias. These data, however, have some limitations including lack of clinical, lifestyle, and demographic data and because of the observational nature, which can introduce biases. These biases include selection and channeling bias, as well as confounding by indication. These limitations can be minimized by careful adjustment in statistical analyses.
In observational studies, the outcomes of interest are often influenced by concurrent or preexisting comorbidities. Comorbidity may be defined as the total burden of illnesses unrelated to the principal diagnosis [
]. It is important to adequately adjust for comorbidities in studies in which comorbidities could act as confounders. Given the large number of comorbidities that may be relevant to a given outcome, controlling for individual comorbidities may not be practical for methodological reasons, including loss of power. It may also be necessary to control for the overall burden of comorbidity, rather than the individual effect of each comorbidity.
For that purpose, a number of comorbidity indices have been developed to measure and weigh the overall burden of comorbidities. Some of these instruments have been developed exclusively for use with administrative data, such as the Elixhauser Index (EI) [
]. These comorbidity indices have been widely used in studies using administrative data to control for the overall burden of comorbidities.
However, given the large number of indices available in the literature and the conceptual and methodological differences among them, researchers and health policy makers wishing to control for comorbidity need guidance in selecting the index most appropriate for their specific study. Although previous studies have compared the validity of comorbidity indices, they were limited by not systematically reviewing all indices available or by not explaining the conceptual and methodological differences between indices [
A systematic review of the Charlson comorbidity index using Canadian administrative databases: a perspective on risk adjustment in critical care research.
]. Our systematic review will guide scientists' choice by reviewing all the indices available, explaining their conceptual and methodological differences, and comparing their construct validity. Because there is no “gold standard” in comorbidity measurement, indices are often validated by measuring how well they are able to predict outcomes related to comorbidity, such as mortality or health care utilization (ie, construct validity) [
Accordingly, our aim was to conduct a systematic review with the following objectives: (1) to identify the different instruments used in administrative data studies to measure comorbidity, (2) to compare the instruments at the conceptual level, that is, to describe how each index was developed and/or adapted for use with administrative data and what concept the index aimed to measure, and (3) to evaluate and compare their ability to predict comorbidity-related outcomes.
2. Methods
2.1 Search strategy
A methodological literature search was conducted as of September 2012, using the Ovid platform to search MEDLINE (MEDLINE(R) In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily, and Ovid OLDMEDLINE(R) from 1946) and EMBASE (from 1980). The year limits were dictated by the scope of the databases. We searched for and combined with the Boolean operator “OR” all relevant subject headings, using the “explosion” function where needed, and keywords in titles and abstracts for the two concepts: “Administrative data” and “Comorbidity index.” We combined these two concepts with the Boolean operator “AND.” We excluded articles that were solely abstracts, comments, conference proceedings, editorials, letters, or news. We included only articles published in English. The titles and abstracts of the articles identified by this search were screened by one investigator (M.Y.) and selected for full-text review if relevant to our objectives. From this initial screen, a list of comorbidity indices potentially used in administrative data was identified. To ensure that we captured all relevant indices and their corresponding validation studies, an additional literature search was performed using the same databases. This involved searching titles and abstracts for specific index names. The same screening process was applied to select articles potentially relevant to our objectives.
2.2 Study selection
Full-length articles of studies identified as potentially relevant to our objectives were independently reviewed by two authors (M.Y. and J.T.) to determine if they met the prespecified inclusion criteria. Disagreements were settled by consensus. For inclusion, studies had to have developed or validated a comorbidity index for use with administrative data. Of note, we only included studies that related specifically to comorbidity indices and excluded studies that focused on the development or adaptation of risk scores or other groupers for risk adjustment. Adaptation of an index initially designed for use in a different context was permitted, such as adaptations of the CCI, which was initially designed for medical chart review. Validation studies could be prospective or retrospective and could include any patient population (adult or pediatric). We defined a validation study as one that evaluated the ability of a comorbidity index to predict a specific outcome (ie, construct validity) in a given population sample. This could be achieved by reporting the C statistic (for dichotomous outcome variables) or the R2 (for linear outcome variables). Alternatively, odds ratios (ORs), relative risk, or hazard ratios (HR) (Cox) could be reported.
2.3 Data abstraction and reporting
Data were abstracted using a standardized data collection form. Abstracted data included the comorbidity indices evaluated, study population, type of administrative data used to calculate the comorbidity score, outcome, and statistics used to evaluate predictive ability. For index development studies, we collected information on the study population, type of data used, and information on how the index was developed or adapted.
For validation studies, we report on construct validity by presenting results of the ability of indices to predict outcomes related to comorbidity, which we have labeled predictive ability. For dichotomous outcomes, such as mortality, the predictive ability was reported using the area under the curve in a receiver operating characteristic curve, which is equivalent to the C statistic, a measure of model fit. The C statistic ranges from no predictive ability (when equaling 0.50) to perfect prediction (when equaling 1.0). Consistent with recommended guidelines [
], we considered C statistics of 0.7–0.8 as acceptable and ≥0.8 as excellent. For continuous outcomes in linear regression models, the predictive ability was measured using the R2 value, which represents the improvement in explained variance obtained by adding the comorbidity score to a baseline model. R2 values range from 0 to 1, where 1 indicates that all the observed variance in the outcome is explained by the model.
3. Results
3.1 Study selection
The primary literature search revealed 565 citations for title and abstract review. The second search, to identify validation studies of the indices identified in the first search, identified 390 additional articles. Thus, a total of 955 articles were selected for title and abstract review (Fig. 1). Of these, 37 were duplicates and 713 were not relevant to the study objectives, leaving 205 articles for full-text review. Of those, 18 studies did not involve the development or validation of an index, 55 involved an index not using administrative data, and 56 discussed a risk score rather than a comorbidity index. Therefore, these 112 studies were excluded, and a total of 76 articles were included in the final review.
Fig. 1Flow chart of studies included in the review.
Comorbidity indices identified were categorized into two groups: (1) those based on diagnoses from administrative data, using International Classification of Disease, Ninth or Tenth revision diagnostic coding system (ICD-9 or ICD-10) and (2) those based on medications, using prescription data to identify comorbid conditions. The 76 articles included 39 studies of diagnosis-based indices, including 35 related to the CCI and its adaptations, two specifically to the EI, and two reporting study-specific diagnosis-based indices. An additional 13 studies investigated medication-based indices such as the Chronic Disease Score (CDS) and the RxRisk. The remaining 24 articles compared the main indices identified.
] in 1987. It was developed using chart review to predict 1-year mortality in a cohort of 604 patients admitted to a medical service at New York Hospital during 1 month in 1984. The CCI was then validated in the same study using a cohort of 685 breast cancer patients admitted to a Connecticut teaching hospital from 1962 to 1969. The final index is a list of 19 conditions, with each condition assigned a weight of 1, 2, 3, or 6 based on adjusted HR for each comorbid condition derived from Cox proportional hazards regression models. A total score is calculated from the sum of the weighted scores [
The CCI is the most widely used comorbidity index and has been validated in patient populations with various diagnoses or undergoing various surgical procedures [
] as described in the following paragraph. For each adaptation, the study populations and primary end points used for development, along with a list of comorbid conditions included, are summarized in Table 1. The results of validation studies are summarized in Table 1/Appendix A at www.jclinepi.com.
Table 1Description of adaptations of the Charlson Comorbidity Index (CCI)
Indicates updated Quan CCI developed in 2011 [39].
Data source, diagnostic codes
Medicare, United States; ICD-9-CM diagnosis and procedure codes
1.
Manitoba Health Services Commission files. ICD-9-CM codes, up to 16 diagnoses
2.
California Office State-wide Health Planning and Developments Hospital discharge data. ICD-9-CM codes, up to 24 secondary diagnoses, up to 25 procedures
Quebec MED-ECHO hospital database. First 3 digits of ICD-9 code without CM, in principal diagnosis, up to 15 secondary diagnoses and up to 5 procedures
Massachusetts Health Data Consortium discharge abstracts. ICD-9-CM codes, up to 15 diagnoses and 15 procedures
Calgary Health Region Hospitalization data. ICD-9-CM (2001–2002) and ICD-10-CA (2002–2003) codes,
] adapted the CCI by identifying the ICD-9 codes corresponding to the 19 original comorbid conditions. The codes for leukemia and lymphoma were combined with the category “any malignancy” leaving the Deyo CCI as a list of 17 comorbid conditions [
]. The Deyo CCI's ability to predict mortality ranged from poor to excellent, with C statistics ranging from 0.64 to 0.86 for in-hospital mortality and 0.59–0.85 for 1-year mortality. A number of studies demonstrated that other indices or risk scores performed better than the Deyo CCI in predicting mortality [
]. The Deyo CCI demonstrated better ability to predict 1-year mortality when both prior inpatient and outpatient data were used to calculate the index [
]. In 2004, the Deyo CCI was modified for use with ICD-10 codes, which performed similarly to the original ICD-9 version in predicting in-hospital mortality [
The Romano CCI, originally known as the Dartmouth-Manitoba CCI, adapted the CCI for use with administrative data. The identification of corresponding International Classification of Disease, Ninth Revision, Clinical Modification (ICD-9-CM) codes was first done by Roos et al. [
] in 1993 to become what is known as the Romano CCI. Compared with the Deyo CCI, the Romano adaptation includes broader definitions, encompassing more ICD-9-CM codes, for peripheral vascular diseases, complicated diabetes, and malignancy. Romano et al. evaluated the Romano and Deyo versions of the CCI in patients who underwent coronary artery bypass grafting (CABG) surgery in Manitoba and lumbar discectomy in California (Table 1). Although no direct comparison of the two adaptations was tested, both demonstrated similar ability to predict outcomes related to comorbidity, when evaluated in the same population and using the same outcome. However, the risk estimate for each comorbidity varied widely across populations. Accordingly, Romano et al. [
]. Roos et al. created an augmented version of the Romano CCI, which demonstrated improved predictive ability compared with the original (Table 1/Appendix A at www.jclinepi.com). However, the authors cautioned that this augmented index may factor in complications resulting in an overestimation of comorbidity [
]. The scores derived from both adaptations in each study demonstrated substantial or almost perfect agreement, indicating that the two comorbidity classifications are similar [
]. Both adaptations demonstrated similar ability to predict mortality (Table 1/Appendix A at www.jclinepi.com); however, the Romano method was slightly superior for predicting mortality, and models with study-derived weights outperformed Charlson weighted models [
]. ICD-10 adaptations of both the Deyo and Romano CCIs demonstrated acceptable predictive ability for 1-year mortality, with the Romano performing slightly better [
D'Hoore et al. created a CCI adaptation using only the first three digits of ICD-9 coding without CM, as many institutions outside the United States use ICD-9 codes without CM (which includes procedural codes and additional morbidity details). Because coding of the tailing digits in ICD-9 codes can lead to inconsistencies, they claim to have created a simpler and more reliable adaptation [
]. The D'Hoore index demonstrated excellent ability to predict in-hospital mortality in populations with a principal diagnosis of myocardial infarction, ischemic heart disease, and bacterial pneumonia but demonstrated poor discrimination in stroke and congestive heart failure populations [
] created a study-specific index with a reduced number of comorbidities, by including only comorbidities found to be associated with in-hospital mortality (OR > 1.2), and with study-specific weights for each comorbidity, derived from multiple logistic regression analyses on the study sample used for index development. The Ghali CCI includes only five comorbidities. When tested on the same sample, it performed better than the Deyo CCI in predicting in-hospital mortality. However, the performance of the Ghali CCI improved further when the coefficients from the original CCI were used instead of the study-specific weights.
] identified the ICD-10 codes corresponding to the Deyo CCI coding algorithm and also expanded the selection of codes for each comorbidity, using physicians to assess the face validity of the selected ICD-10 codes.
], which is not included in our review because the adaptation was not validated. In cohorts from four countries, the Quan CCI was a better predictor of in-hospital mortality than the Sundararajan and Halfon adaptations [
] compared the ability of the Quan CCI and adjusted clinical groups (ACGs) to predict medication use and found that the ACGs predicted better. Of note, ACGs use ICD-9 or ICD-10 codes to develop a composite measure of patient illness burden, estimated from the mix of conditions experienced for a defined interval.
] created an updated version of their index and derived study-specific weights in a similar method to Ghali's. The updated index includes 12 comorbidities with new weights assigned to each [Table 1] and demonstrated slightly better ability to predict in-hospital, 30-day, and 1-year mortality than the original Quan CCI [
Martins and Blais developed a study-specific index including 23 comorbidities, eight from the original Charlson index and 13 identified as frequent comorbidities in the study population. This index demonstrated superior performance for predicting in-hospital mortality compared with a CCI adaptation (specific adaptation not specified) [
Klabunde et al. created two new indices referred to as the National Cancer Index (NCI)—one using inpatient claims and the other using outpatient claims—by assigning weights to the comorbidities from the original Charlson index based on a Cox proportional hazards model predicting 2-year noncancer mortality. It was evaluated for its ability to predict future treatment in both prostate and breast cancer populations, demonstrating acceptable and excellent predictive ability, respectively [
]. The index was revised by combining the score from inpatient and outpatient data and was further evaluated, compared with the original version, a uniform weights index and the CCI (specific adaptation not specified), for its ability to predict 2-year noncancer mortality in four cohorts with breast, prostate, colorectal, and lung cancer. The new NCI demonstrated similar predictive ability to the original NCI, and both versions performed better than the CCI or the uniform weight index in all four study populations [
3.2.1.7 Comparison of self-report vs. administrative data-derived CCI
Three studies compared the predictive ability of CCI adaptations derived from self-reported data with the same index derived from administrative data [
Comorbidity information in older patients at an emergency visit: self-report vs. administrative data had poor agreement but similar predictive validity.
Comorbidity information in older patients at an emergency visit: self-report vs. administrative data had poor agreement but similar predictive validity.
] found that self-report of comorbid conditions had varying levels of agreement with those derived from administrative data, ranging from poor to substantial agreement depending on the comorbid condition (κ = 0.14–0.79).
3.2.1.8 Comparison of chart review vs. administrative data-derived CCI
] conducted a systematic review of studies comparing CCI adaptations derived from administrative data and chart review. They found that CCI scores calculated from administrative data were consistently lower than those derived from chart review, and agreement between the two sources was poor to fair (κ ranging from 0.30 to 0.56) [
A comparison of the Charlson comorbidity index derived from medical records and claims data from patients undergoing lung cancer surgery in Korea: a population-based investigation.
], evaluating a study-specific administrative data CCI using ICD-10 codes, found CCI scores derived from the two sources to be well correlated (r = 0.88, P < 0.01) [
Elixhauser et al. developed a comorbidity index comprised of a comprehensive set of 30 comorbidities defined using ICD-9-CM codes from administrative data (Table 2/Appendix B at www.jclinepi.com). The EI comorbidities were significant predictors of LOS and hospital charges. Many of the individual EI comorbidities were associated with in-hospital mortality, but as a group, the association was not significant [
]. One disadvantage of the original EI is that it includes 30 dichotomous variables, representing each comorbidity, without a weighting system to provide a single score.
Three studies validated the EI for its ability to predict mortality, with two providing a modification to the EI [
]. Additional studies have validated and compared the EI with other indices (Section 3.5). In predicting in-hospital mortality, all EI versions demonstrated acceptable-to-excellent predictive ability [
]. Van Walraven et al. developed a scoring system for the EI, using the regression coefficients for each comorbidity from a multivariate logistic regression model predicting in-hospital mortality. It demonstrated acceptable prediction of in-hospital mortality, with similar results for models using the EI including all comorbidities and the EI including only the 21 comorbidities significantly associated with mortality [
]. Using hospitalization and outpatient data from the Surveillance, Epidemiology, and End Results—Medicare linked database, Fleming et al. developed an index with 27 comorbidity categories based on the prevalence of diseases specifically in their study population, black males with prostate cancer [
] created a comorbidity index based on administrative data to predict 30-day mortality after CABG. Their index performed similarly to the additive EuroSCORE registered in a clinical database.
3.4 Diagnosis-based indices comparison studies
Fourteen studies compared the EI to various CCI adaptations [
Comparison of the performance of the CMS Hierarchical Condition Category (CMS-HCC) risk adjuster with the Charlson and Elixhauser comorbidity measures in predicting mortality.
Using the Johns Hopkins' Aggregated Diagnosis Groups (ADGs) to predict 1-year mortality in population-based cohorts of patients with diabetes in Ontario, Canada.
]. A description of the cohorts and results for each study are found in Table 3/Appendix C at www.jclinepi.com. Overall, both the EI and CCI demonstrated poor-to-excellent ability to predict various outcomes. When predicting in-hospital mortality, C statistics ranged from 0.632 to 0.878 and 0.608–0.860 for the EI and CCI, respectively. For 1-year mortality, results ranged from 0.69 to 0.909 and 0.65–0.906, respectively. Nine studies [
] demonstrated that various versions of the EI (including the original, Quan, and van Walraven adaptations) predicted mortality outcomes better than various adaptations of the CCI (Deyo, Romano, and Quan adaptations). In contrast, four studies found no difference [
Comparison of the performance of the CMS Hierarchical Condition Category (CMS-HCC) risk adjuster with the Charlson and Elixhauser comorbidity measures in predicting mortality.
]. Three studies examined the effect of using inpatient and/or outpatient data on predicting mortality and found that combining data from both sources resulted in higher C statistics [
Comparison of the performance of the CMS Hierarchical Condition Category (CMS-HCC) risk adjuster with the Charlson and Elixhauser comorbidity measures in predicting mortality.
]. Lieffers et al. augmented the EI by adding performance status and substituting clinical data instead of administrative records for body weight. This augmented version of EI predicted 2- and 3-year survival in patients with stages II–IV colorectal cancer better than the Quan EI. Another study evaluated a combination of the van Walraven EI and the Romano CCI and found the combined index predicted mortality better than each individual index [
Medication-based indices use pharmacy data to identify comorbidities by linking medications to specific disease categories. Fourteen studies examined the development or validation of these indices [
] created the CDS, using medications instead of diagnostic codes to identify comorbidities. Using a population-based pharmacy database, a panel of experts evaluated patterns of use of selected medications to create disease categories, and weights were assigned by consensus [
] modified the original CDS by updating medications, expanding the disease categories to 28, and weighting the disease categories based on regression models. The CDS-2 ubiquitously replaced the original CDS, and adaptations were subsequently made for application to specific populations, including the Pediatric Chronic Disease Score [
] continued updating the CDS-2 to include new medication classes and expand disease categories, ultimately developing a new but related instrument, RxRisk.
3.7 RxRisk and RxRisk-V
The RxRisk was developed as an all-age risk assessment instrument using outpatient pharmacy data to identify chronic diseases and predict future health care costs [
]. The RxRisk included 57 adult and pediatric weighted disease categories and associated drug classes. Validation was based on a large general population sample using multiple measures of predictive power. The RxRisk-V was a subsequent modification adapted to the Veterans Health Administration population [
The final medication-based index is the Medication-Based Disease Burden Index (MDBI), developed as an alternative to the original CDS to deal with the same issues addressed by Clark's and Fishman's revisions [
Utility of the chronic disease score and Charlson comorbidity index as comorbidity measures for use in epidemiologic studies of antibiotic-resistant organisms.
]. Schneeweiss et al. conducted three studies comparing medication- and diagnosis-based indices for their ability to predict various outcomes (Table 4/Appendix A at www.jclinepi.com). Two studies found the following performance ranking when predicting 1-year mortality, long-term care admissions, and hospitalizations: Romano CCI ≥ Deyo CCI > D'Hoore CCI > Ghali CCI > CDS-1 > CDS-2, but a different ranking when predicting physician visits or expenditures for physician visits: D'Hoore CCI > CDS-2 ≥ Romano CCI > Deyo CCI > CDS-1 > Ghali CCI [
]. Another study evaluated a study-specific adaptation of the Romano CCI, which derived its own study-specific weights and found that it predicted 1-year mortality better than the original Romano CCI and that both versions outperformed the CDS-1. However, the EI demonstrated the best predictive ability of the four indices compared [
Utility of the chronic disease score and Charlson comorbidity index as comorbidity measures for use in epidemiologic studies of antibiotic-resistant organisms.
], the predictive ability of the diagnosis-based index (Deyo CCI) was better than the medication-based index (CDS-1); yet, another study demonstrated the opposite [
]. Generally, medication-based indices demonstrated better ability to predict various health care utilization outcomes, including prescription medication use [
In this report, we summarize the published literature on the development and validity of comorbidity indices used in administrative data studies. The body of literature on this topic is broad, as we identified a total of 76 primary articles for inclusion.
All indices identified could be grouped as either diagnosis based, using ICD coding, or medication based, using pharmacy dispensing data. The main diagnosis-based indices were the EI and the various adaptations of the CCI for use with administrative data. Medication-based indices included versions of the CDS, which later became known as the RxRisk, and its adaptation for use in the veteran population, the RxRisk-V.
Of the diagnosis-based measures, we found that the EI consistently outperformed the CCI in predicting both short- and long-term mortality. Of the main adaptations of the CCI, the Romano CCI demonstrated equal or better performance in its ability to predict various outcomes compared with the Deyo CCI, despite the fact that the Deyo CCI is the more commonly used measure. Although both the EI and all administrative data adaptations of the CCI were developed for use with inpatient hospitalization data, several studies found that using combination of both inpatient and outpatient data consistently improved the performance of the index studied. Furthermore, a number of studies examined adaptations of the CCI and EI, which derived empirical weights based on the study-specific population and outcome measure. Assigning study-specific weights for both EI and CCI adaptations tended to improve their predictive performance. Accordingly, for diagnosis-based indices, we recommend the use of the EI or the Romano CCI, particularly when predicting mortality outcomes. When available, we recommend calculating these indices using a combination of both inpatient and outpatient data and, when possible, deriving study-specific empirically derived weights for the index selected.
Of the medication-based indices, we found that the original version of the CDS developed by von Korff et al. [
]. However, the later version known as the RxRisk-V was the most commonly used medication-based measure and demonstrated the best predictive ability. Thus, in studies using pharmacy dispensing data, we recommend use of the RxRisk-V.
In studies comparing the predictive ability of indices, we found that diagnosis-based measures were better predictors of mortality outcomes than medication-based indices. Some studies found that medication-based indices were better predictors of health care utilization and costs; however, other studies found that diagnosis-based measures were better at predicting such outcomes. Disadvantages of diagnosis-based indices include a wide variability in ICD coding practices, underreporting of chronic conditions in the secondary diagnosis fields, and difficulty in distinguishing between acute conditions present on admission from subsequent complications of care [
A systematic review of the Charlson comorbidity index using Canadian administrative databases: a perspective on risk adjustment in critical care research.
]. Furthermore, certain CCI adaptations can only be used with specific ICD versions (eg, Deyo CCI with ICD-9-CM and the Quan CCI with ICD-10). Additionally, a number of country-specific ICD-10 versions exist [
], which may further limit the application of the diagnosis-based indices. Hence, it is important, when choosing a CCI adaptation, to consider the ICD version used in the administrative data of the study and to select the CCI index accordingly. Pharmacy data are credited as a more timely, complete, and reliable data source than diagnosis-based data [
], but it is not readily available in many jurisdictions. The major criticism of medication-based indices lies in the back coding from prescription to diagnosis [
]. This limits the definition of comorbidity to include only chronic diseases treated with prescription medication. Furthermore, medication-based indices require continual updates to accommodate the development and reassignment of new medications for specific indications.
We found that the predictive ability of comorbidity indices varied widely, ranging from poor (C statistic 0.50–0.69) to good (C statistic 0.70–0.79) and excellent (C statistic >0.80), with similar variability observed when R2 values were reported. Performance varied according to the specific index, outcome measured, and study population. In 2000, Schneeweiss et al. [
] conducted the first review of studies evaluating the ability of comorbidity indices to predict comorbidity-related outcomes, using administrative data. They concluded that comorbidity indices using administrative data provide only a modest improvement over adjustment for age alone [
]. In 2001, they conducted an extensive validation study of four adaptations of the CCI as well as the CDS-1 and CDS-2. They concluded that comorbidity indices provide only a limited ability to control for confounding, acknowledging nonetheless their usefulness because of their ease of use and time and resources savings [
]. In 2005, Needham et al. conducted a review of 10 articles on CCI adaptations for administrative data with a specific emphasis on risk adjustment in critical care research. They found no difference in mortality prediction whether using CCI derived from administrative data or chart review and with the various adaptations of the CCI [
A systematic review of the Charlson comorbidity index using Canadian administrative databases: a perspective on risk adjustment in critical care research.
]. The authors performed a meta-analysis examining the indices' performance in predicting short- and long-term mortality. Their extensive comparative analysis resulted in findings similar to those in our study; specifically, the EI and Romano CCI demonstrated significantly superior performance in predicting mortality outcomes. Although our study and that by Sharabiani et al. focus on comorbidity indices using administrative data, they examine different aspects of the topic and use distinct approaches. Our study provides an extensive description of the various indices available, including how they were developed and results of their validation studies, and outlines the differences between them. The aforementioned review only includes diagnosis-based indices and reports only on studies comparing the predictive performance of various indices. Therefore, they did not discuss medication-based indices or include studies comparing medication- and diagnosis-based indices. Our study offers complementary information valuable to researchers trying to understand the differences between available instruments. It will assist researchers in selecting a comorbidity index that best meets the needs of their specific study, including when administrative data are available on both diagnosis and medication information.
There are, however, limitations to our systematic review. First, we chose to focus specifically on comorbidity indices using administrative data and did not evaluate comorbidity indices using chart review or self-reported data. Predictive validity of CCI adaptations may differ with these data sources. Second, although our search strategy was comprehensive, it is possible that studies were missed. However, given the consistency of our results, it is unlikely that missed studies would significantly alter the main findings of our review. Finally, the number of published comparison studies between medication- and diagnosis-based indices was limited, and results were not entirely consistent across studies.
5. Conclusion
Comorbidity indices are used to control for the overall burden of comorbidities in administrative data studies and demonstrate validity in predicting mortality; however, their ability to fully adjust for confounding due to comorbidity may be limited. We recommend using a diagnosis-based index, such as the Quan EI, van Walraven EI, or Romano CCI, in studies in which the outcome of interest is mortality. One must consider the ICD version used when selecting a specific index. For studies evaluating health care utilization, in which medication data are available, we recommend using a medication-based index, such as the RxRisk-V. Overall, the appropriate selection of a comorbidity index for use with administrative data should take into account the type of data available, the study population, and the specific outcome of interest in the study.
A systematic review of the Charlson comorbidity index using Canadian administrative databases: a perspective on risk adjustment in critical care research.
Comorbidity information in older patients at an emergency visit: self-report vs. administrative data had poor agreement but similar predictive validity.
A comparison of the Charlson comorbidity index derived from medical records and claims data from patients undergoing lung cancer surgery in Korea: a population-based investigation.
Comparison of the performance of the CMS Hierarchical Condition Category (CMS-HCC) risk adjuster with the Charlson and Elixhauser comorbidity measures in predicting mortality.
Using the Johns Hopkins' Aggregated Diagnosis Groups (ADGs) to predict 1-year mortality in population-based cohorts of patients with diabetes in Ontario, Canada.
Utility of the chronic disease score and Charlson comorbidity index as comorbidity measures for use in epidemiologic studies of antibiotic-resistant organisms.
00357-6&id=gr1.jpg){kind=link}