Abstract
Objective
Some previously developed risk scores contained a mathematical error in their construction: risk ratios were added to derive weights to construct a summary risk score. This study demonstrates the mathematical error and derived different versions of the Charlson comorbidity score (CCS) using regression coefficient–based and risk ratio–based scoring systems to further demonstrate the effects of incorrect weighting on performance in predicting mortality.
Study Design and Setting
This retrospective cohort study included elderly people from the Clinical Practice Research Datalink. Cox proportional hazards regression models were constructed for time to 1-year mortality. Weights were assigned to 17 comorbidities using regression coefficient–based and risk ratio–based scoring systems. Different versions of CCS were compared using Akaike information criteria (AIC), McFadden's adjusted R2, and net reclassification improvement (NRI).
Results
Regression coefficient–based models (Beta, Beta10/integer, Beta/Schneeweiss, Beta/Sullivan) had lower AIC and higher R2 compared to risk ratio–based models (HR/Charlson, HR/Johnson). Regression coefficient–based CCS reclassified more number of people into the correct strata (NRI range, 9.02–10.04) compared to risk ratio–based CCS (NRI range, 8.14–8.22).
Conclusion
Previously developed risk scores contained an error in their construction adding ratios instead of multiplying them. Furthermore, as demonstrated here, adding ratios fail to even work adequately from a practical standpoint. CCS derived using regression coefficients performed slightly better than in fitting the data compared to risk ratio–based scoring systems. Researchers should use a regression coefficient–based scoring system to develop a risk index, which is theoretically correct.
Keywords
What is new?
- •Theoretically, a scoring system derived on an additive scale (i.e., regression coefficient) requires additive weights; it is incorrect to add weights in a system derived on a multiplicative scale (i.e., risk ratio). Nevertheless, previous studies have developed risk scores adding odds/hazards ratio weights to construct a summary score.
- •Regression coefficient-based models had lower AIC and higher R2 compared to risk-ratio based models.
Key findings
- •This study demonstrated the mathematical error and compared different versions of the Charlson comorbidity score derived using regression coefficient–based or risk ratio–based scoring systems. Results showed that the correct use of the regression coefficient–based scoring system performs slightly better than the incorrect use of a risk ratio–based scoring system in predicting mortality.
What this adds to what was known?
- •The scoring system used affects the performance of a comorbidity score. Researchers should use a mathematically correct regression coefficient–based scoring system when developing a comorbidity score or risk index.
What is the implication and what should change now?
1. Introduction
Summary comorbidity measures, such as the Charlson comorbidity score (CCS), condense the overall burden of illness into a single numeric score that can be used for confounding control or prognostic assessment [
1
, 2
, 3
, 4
]. As per a Web of Science report, the original Charlson article has been cited more than 10,000 times, indicating that the use of such measures has increased substantially in past 2 decades [3
, 5
]. To derive summary comorbidity measures, the usual approach is to develop a regression model (linear, logistic, or survival) for an outcome of interest while including age, gender, and baseline comorbidities as independent variables. Weights would be assigned to individual comorbidities using a scoring algorithm which is based on regression coefficients, risk ratio, or clinical judgment. Weights would be summed for a particular patient to obtain a single numeric score, referred to as summary comorbidity scores.Scoring algorithms based on regression coefficient or risk ratio have been used to assign weights to different comorbidity scores such as CCS, Elixhauser comorbidity score, chronic disease score, Rx-Risk-V, HRQoL-comorbidity index, and combined comorbidity score [
1
, 4
, 5
, 6
, 7
, 8
, 9
]. Some previously developed risk scores contained a mathematical error in their construction by using a risk ratio–based scoring system. Harrell first pointed out in 1996 that weights for CCS should have been assigned based on the regression coefficient rather than the risk ratio because risk ratios do not add but multiply, whereas regression coefficients can be added. Charlson replied by stating that “whether or not alteration of the scaling would improve its usefulness is certainly an issue that could be empirically addressed” [[10]
]. The issue was again raised in 2002 by Moons et al., who asked, “Should scoring rules be based on odds ratios or regression coefficients?” [[11]
].Mathematically, a scoring system developed based on an additive scale, that is, regression coefficients, is correct than a system based on a multiplicative scale, that is, odds ratio (OR) or hazard ratio [
10
, 11
, 12
]. However, no empirical evidence was ever generated to test Charlson's reply. Therefore, the goal of this study was to derive different versions of CCS using regression coefficient–based and risk ratio–based scoring systems and compare their performance in predicting 1-year mortality.2. Methods
2.1 Clinical Practice Research Datalink
The study used the Clinical Practice Research Datalink (CPRD) database, an electronic medical record data from the United Kingdom [
13
, 14
]. This retrospective longitudinal cohort study included patients 65 years and older who were continuously enrolled in the baseline year 2008. Clinical and referral claims from the baseline year were used to construct CCS. All patients were followed up for 1 year (i.e., from January 1, 2009 to December 31, 2009) to observe mortality.2.2 Development of Charlson comorbidity score
Baseline clinical and referral claims were queried for the presence of 17 Charlson disease conditions which were coded as yes or no [
[15]
]. A multivariate Cox proportional hazards regression model was constructed for time to 1-year mortality while including age, gender, and 17 Charlson disease conditions as independent variables. Table 1 reports the regression coefficient and hazard/odds ratio–based scoring algorithms used to assign weights to 17 Charlson disease conditions. All weights were summed to construct a summary CCS. In addition, prior studies have derived weights for Charlson comorbidity conditions using different data sets (Table 2). We used these in the present study for comparison purpose.Table 1Regression coefficient–based and hazard/odds ratio–based scoring algorithms to derive weights for comorbidity score
| Weighing scheme | Scoring algorithm to derive weights for comorbidity score |
|---|---|
| Regression coefficient–based scoring algorithms | |
| Beta | Use the original beta value |
| Beta/integer | The beta coefficients were rounded to the nearest integer to derive weights |
| Beta10/integer | The beta coefficients were multiplied by 10 and rounded to the nearest integer to derive weights |
| Beta/Schneeweiss (Schneeweiss's scoring system) [16] | Weights were increased by 1 unit with each 0.3 increase in the β. Therefore, a weight of 1 refers to an e(0.30) = 35% increase in risk of dying during the follow-up year. −1.05 > β ≥ −1.35 then weight = −4 −0.75 > β ≥ −1.05 then weight = −3 −0.45 > β ≥ −0.75 then weight = −2 −0.15 > β ≥ −0.45 then weight = −1 0 > β ≥ −0.15 then weight = 0 0 ≤ β ≤ 0.15 then weight = 0 0.15 < β ≤ 0.45 then weight = 1 0.45 < β ≤ 0.75 then weight = 2 0.75 < β ≤ 1.05 then weight = 3 1.05 < β ≤ 1.35 then weight = 4 1.35 < β ≤ 1.65 then weight = 5 1.65 < β ≤ 1.95 then weight = 6 1.95 < β ≤ 2.25 then weight = 7 |
| Beta/Sullivan (Sullivan's scoring system) [12] | The beta coefficients were divided by the smallest absolute value of regression coefficient and rounded to the nearest integer. |
| Hazard/odds ratio–based scoring algorithms | |
| HR/Charlson (Charlson's scoring system) 1 , 9 | 1.0 < HR < 1.5 then weight = 1 1.5 ≤ HR < 2.5 then weight = 2 2.5 ≤ HR < 3.5 then weight = 3 3.5 ≤ HR < 4.5 then weight = 4 4.5 ≤ HR < 6 then weight = 5 If HR ≥ 6.0 then weight = 6 |
| OR/Johnson (Johnson's scoring system) [8] | OR < 1 then weight = −1 1.0 ≤ OR < 1.2 then weight = 1 1.2 ≤ OR < 1.4 then weight = 2 1.4 ≤ OR < 1.6 then weight = 3 1.6 ≤ OR < 1.8 then weight = 4 1.8 ≤ OR < 2.0 then weight = 5 If OR ≥ 2.0, then weight = 6 For any OR, if P > 0.10, then weight = 0 |
Abbreviations: CCS, Charlson comorbidity score; CPRD, Clinical Practice Research Datalink; HR, hazards ratio; OR, odds ratio.
a Quan et al. used same scoring algorithm.
Table 2Different versions of the Charlson comorbidity score
| Charlson comorbidity score | Derivation data source | Validation data source | Outcome | Weights derivation algorithm | Number of disease conditions receiving nonzero weights |
|---|---|---|---|---|---|
| Charlson, 1987 [1] | All patients admitted to New York Hospital-Cornell Medical Center | Breast cancer patients admitted to Yale New Haven hospital | One-year mortality | Hazard ratio scoring system | 17 |
| Schneeweiss, 2003 [16] | New Jersey Medicare data | Pennsylvania Medicare data | One-year mortality | Regression coefficient scoring system | 15 |
| Quan, 2011 [9] | Inpatient hospital data from Calgary Health Region, Canada | Hospital data from six different countries | One-year mortality | Hazard ratio scoring system | 12 |
2.3 Comparative performance of Charlson comorbidity scores
Descriptive statistics were used to describe the baseline characteristics and comorbidity scores in CPRD data. Logistic regression models were constructed with 1-year mortality as the dependent variable. A total of 12 models were constructed which included the baseline (age + gender) model and 11 models based on different versions of CCS. All models were compared using the following metrics to determine which scoring system based models fit the data well [
17
, 18
, 19
, 20
, 21
]. We used Akaike information criteria (AIC) to compare regression coefficient–based and odds ratio–based models. AIC compare models based on their fit to the data but give penalty to the complex models. Adjusted Macfadden R2 was used as a goodness-of-fit measure. The lower value of AIC and higher value of R2 indicate good fit. Reclassification measure such as net reclassification improvement (NRI) was used to determine clinical usefulness [22
, 23
]. The NRI assesses risk reclassification of cases and controls into correct risk strata. Positive and significant NRI value suggests that the new model classified patients into the correct risk strata compared to the old model.All statistical analyses were performed using SAS 9.4 (SAS Inc., Cary, NC, USA) and STATA 13 (Stata Corporation, College Station, TX, USA).
3. Results
The CPRD study cohort included 7,66,208 elderly people. The mean age was 75 years (standard deviation = 8) and nearly half of the people were males. Frequencies of Charlson comorbidity conditions and distribution of CCS are reported in Appendix A, Appendix B at www.jclinepi.com, respectively.
Table 3 reports regression coefficients, hazard ratios, and weights for 17 Charlson disease conditions. Regression coefficient and hazard ratios were used to derive weights for 17 disease conditions based on the different algorithms mentioned in Table 1. CCS weights from previous studies are also included in Table 3, that is, CCS Original, CCS Schneeweiss, and CCS Quan.
Table 3Deriving weights for Charlson comorbidity diseases using different scoring systems
| Charlson disease category | Beta estimate (β) | Hazards ratio | Regression coefficient–based weights | Hazards/odds ratio–based weights | Charlson weights from prior studies | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Beta/integer | Beta10/integer | Beta/Schneeweiss | Beta/Sullivan | HR/Charlson | HR/Johnson | CCS Original [1] | CCS Schneeweiss [9] | CCS Quan [20] | |||
| Myocardial infarction | 0.55 | 1.73 | 1 | 6 | 2 | 11 | 2 | 4 | 1 | 1 | 0 |
| Congestive heart disease | 0.93 | 2.54 | 1 | 9 | 3 | 19 | 3 | 6 | 1 | 2 | 2 |
| Peripheral vascular disease | 0.59 | 1.80 | 1 | 6 | 2 | 12 | 2 | 5 | 1 | 1 | 0 |
| Cerebrovascular disease | 0.46 | 1.58 | 1 | 5 | 2 | 9 | 2 | 3 | 1 | 1 | 0 |
| Dementia | 0.77 | 2.15 | 1 | 8 | 3 | 15 | 2 | 6 | 1 | 3 | 2 |
| Chronic pulmonary disease | 0.08 | 1.08 | 0 | 1 | 0 | 2 | 1 | 1 | 1 | 2 | 1 |
| Rheumatological disease | 0.05 | 1.05 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 1 |
| Peptic ulcer disease | 0.59 | 1.80 | 1 | 6 | 2 | 12 | 2 | 4 | 1 | 0 | 0 |
| Mild liver disease | 0.91 | 2.48 | 1 | 9 | 3 | 18 | 2 | 6 | 1 | 2 | 2 |
| Diabetes | 0.12 | 1.13 | 0 | 1 | 0 | 2 | 1 | 2 | 1 | 1 | 0 |
| Diabetes with complications | 0.23 | 1.26 | 0 | 2 | 1 | 5 | 1 | 2 | 2 | 1 | 1 |
| Hemiplegia | 0.83 | 2.28 | 1 | 8 | 3 | 17 | 2 | 6 | 2 | 3 | 2 |
| Renal disease | 0.23 | 1.25 | 0 | 2 | 1 | 5 | 1 | 2 | 2 | 2 | 1 |
| Cancer | 1.20 | 3.33 | 1 | 12 | 4 | 24 | 3 | 6 | 2 | 2 | 2 |
| Moderate liver disease | 1.58 | 4.84 | 2 | 16 | 5 | 32 | 5 | 6 | 3 | 4 | 4 |
| Metastatic tumor | 2.30 | 9.93 | 2 | 23 | 8 | 46 | 6 | 6 | 6 | 6 | 6 |
| AIDS | 6 | 4 | 4 | ||||||||
| Maximum possible comorbidity score | — | — | 11 | 114 | 39 | 230 | 36 | 65 | 33 | 35 | 28 |
Abbreviation: CCS, Charlson comorbidity score.
a AIDS was not included because of very low sample size.
Table 4 reports the comparison of the regression coefficient and the risk ratio–based scoring systems. All summary CCS was associated with a higher risk of mortality. The magnitude of OR was different for summary CCS due to the difference in scaling of the summary CCS. The baseline model that included age and gender had the highest AIC (2,32,864) and the lowest R2 (0.092). The model which included 17 indicator variables for Charlson comorbidities performed better than the baseline model (AIC = 2,26,254; R2 = 0.117). Models which included the summary CCS performed better than the baseline model. Regression coefficient–based models (Beta, Beta10/integer, Beta/Schneeweiss, Beta/Sullivan) had lower AIC and higher R2 compared to risk ratio–based models (HR/Charlson; HR/Johnson), suggesting that models based on a regression coefficient scoring system fit data better than models based on an odds ratio–based scoring system. Compared to the baseline model, the regression coefficient–based CCS reclassified 7.79% to 10.04% of patients into the correct risk strata, whereas risk ratio–based CCS reclassified 8.14% to 8.22% of patients. Different metrics showed that CCS derived using beta coefficients with an exception of beta/integer performed slightly better compared to CCS derived using risk ratios. Furthermore, different versions of CCS derived in this study performed better than existing CCS, that is, CCS original, CCS Schneeweiss, and CCS Quan.
Table 4Comparison of regression coefficient–based and risk ratio–based scoring system in the CPRD data
| OR (95% CI) | AIC | McFadden's adjusted R2 | NRI (%) | |
|---|---|---|---|---|
| Logistic regression model | ||||
| Baseline (age + gender) | — | 2,32,864 | 0.092 | Ref |
| CCS categories | — | 2,26,254 | 0.117 | 8.94 |
| Regression coefficient–based scoring system | ||||
| Beta | 3.18 (3.10–3.26) | 2,26,250 | 0.117 | 9.43 |
| Beta/integer | 2.90 (2.83–2.97) | 2,27,118 | 0.114 | 7.79 |
| Beta10/integer | 1.12 (1.12–1.13) | 2,26,257 | 0.117 | 10.04 |
| Beta/Schneeweiss | 1.40 (1.39–1.41) | 2,26,330 | 0.117 | 9.02 |
| Beta/Sullivan | 1.06 (1.06–1.06) | 2,26,253 | 0.117 | 9.46 |
| Hazard/odds ratio–based scoring system | ||||
| HR/Charlson | 1.21 (1.20–1.21) | 2,27,237 | 0.113 | 8.14 |
| HR/Johnson | 1.46 (1.45–1.48) | 2,26,788 | 0.115 | 8.22 |
| Existing CCS | ||||
| CCS Original | 1.47 (1.45–1.48) | 2,28,490 | 0.109 | 6.95 |
| CCS Schneeweiss | 1.43 (1.42–1.45) | 2,28,267 | 0.109 | 6.47 |
| CCS Quan | 1.69 (1.67–1.71) | 2,27,235 | 0.113 | 6.97 |
Abbreviations: CPRD, Clinical Practice Research Datalink; OR, odds ratio; CI, confidence interval; AIC, Akaike information criteria; NRI, net reclassification improvement; CCS, Charlson comorbidity score.
a OR for summary comorbidity score is reported.
b Baseline model was taken as a reference group to calculate NRI values. All NRI values were statistically significant.
c All models were adjusted for age and gender.
4. Discussion
The present study showed that use of a regression coefficient–based vs. risk ratio–based scoring system can alter the performance of the comorbidity score. Different versions of CCS derived using a mathematically correct regression coefficient–based scoring algorithm (except beta/integer) performed slightly better than CCS derived using a risk ratio–based scoring algorithm.
The slightly better performance of a regression coefficient–based scoring system is due to the use of a correct mathematical approach. Moreover, use of a regression coefficient–based scoring system can estimate a patient's risk correctly, whereas a risk ratio–based scoring system can rank a patient's risk inappropriately. Developing a risk score based on a risk ratio–based scoring system and obtaining an individual patient's risk by adding scores is mathematically incorrect and may produce a poor model fit. Our data demonstrate this by comparing different scoring systems. Different versions of CCS developed using a regression coefficient–based scoring system had better model fit than those using a risk ratio–based scoring system.
Use of a risk ratio–based scoring system to assign weights to the comorbidity score can lead to some conceptual and mathematical problems. A protective risk factor (negative regression coefficient) may be made harmful on antilogging. Charlson's scoring system did not report an algorithm for protective risk factor because there was no protective risk factor. In Johnson's algorithm, negative 1 point was given to the risk factor to show protective effect. However, risk factors with lower protective effect may receive equal weight to risk factors with higher protective effect. For example, a risk factor with a −0.10 coefficient will receive a weight of −1 as a risk factor with a −0.90 coefficient . If a continuous risk factor is modeled as a quadratic function, one cannot assign weights based on the odds ratio of a linear or quadratic term; it is mathematically and conceptually not correct to antilog odds ratios before summing up the regression coefficients for a particular value of the risk factor. Antilogs could possibly have been added only if the regression coefficients had represented a log of log of ratios but that would have placed strange restrictions on the effects of risk factors. In addition, the risk index developed based on odds or hazards ratio will give the wrong risk score and incorrectly predict an outcome.
Among regression coefficient–based scoring algorithms, beta/integer did not perform well compared to the other four scoring systems because beta/integer did not capture the difference between different comorbid conditions. For instance, beta coefficients with a value of 0.55 (myocardial infarction) or 0.93 (congestive heart failure) received an equal weight of 1. CCS weights derived for CPRD data performed better compared to weights derived in previous studies. This could be because we used the same data for deriving weights and comparing different versions of CCS. Comorbidity score weights derived for a specific population and data set will obviously perform better than weights derived in other data sets [
16
, 24
]. Future research studies that use CPRD data to study mortality can use CCS weights derived in the present study. In this study, we only used goodness-of-fit measures to compare different scoring system based models. We did not use discrimination-based measures such as c-statistics because it is a less sensitive measure and may not be able to distinguish between different scoring systems [25
, 26
].Previously developed risk scores contained an error in their construction adding ratios instead of multiplying them. Furthermore, as demonstrated here, adding ratios fail to even work adequately from a practical standpoint. Different versions of CCS derived using regression coefficients, with the exception of beta/integer, performed slightly better than those derived using a risk ratio–based scoring system. Researchers should use the regression coefficient–based scoring system as it is mathematically correct, easy to implement, and performs better in fitting the data.
Appendix.
Appendix AFrequency of independent variables among 7,66,208 patients aged ≥65 years from Clinical Practice Research Datalink
| Variables | Number (percentages) |
|---|---|
| Male | 3,36,135 (43.87) |
| Age, mean (SD) | 75 (8) |
| Charlson disease category | |
| Myocardial infarction | 3,295 (0.43) |
| Congestive heart disease | 5,287 (0.69) |
| Peripheral vascular disease | 5,363 (0.70) |
| Cerebrovascular disease | 8,811 (1.15) |
| Dementia | 4,980 (0.65) |
| Chronic pulmonary disease | 13,409 (1.75) |
| Rheumatological disease | 6,743 (0.88) |
| Peptic ulcer disease | 1,226 (0.16) |
| Mild liver disease | 306 (0.04) |
| Diabetes | 60,684 (7.92) |
| Diabetes with complications | 6,360 (0.83) |
| Hemiplegia | 77 (0.01) |
| Renal disease | 39,153 (5.11) |
| Cancer | 22,373 (2.92) |
| Moderate liver disease | 153 (0.02) |
| Metastatic tumor | 843 (0.11) |
| AIDS | 0 (0.00) |
Abbreviation: SD, standard deviation.
Appendix BDistribution of Charlson comorbidity scores
| Charlson comorbidity score | Mean (SD) | Median | % With 0 | Minimum | Maximum |
|---|---|---|---|---|---|
| Beta | 0.09 (0.27) | 0 | 80.06 | 0 | 4.68 |
| Beta/integer | 0.07 (0.28) | 0 | 93.53 | 0 | 5 |
| Beta10/integer | 0.85 (2.71) | 0 | 80.73 | 0 | 47 |
| Beta/Schneeweiss | 0.28 (0.94) | 0 | 88.32 | 0 | 16 |
| Beta/Sullivan | 1.74 (5.47) | 0 | 80.06 | 0 | 94 |
| CCS/Charlson | 0.34 (0.85) | 0 | 80.06 | 0 | 13 |
| HR/Johnson | 0.65 (1.66) | 0 | 80.73 | 0 | 22 |
| CCS Original | 0.32 (0.77) | 0 | 80.06 | 0 | 13 |
| CCS Schneeweiss | 0.34 (0.82) | 0 | 80.83 | 0 | 12 |
| CCS Quan | 0.18 (0.56) | 0 | 87.98 | 0 | 11 |
Abbreviations: SD, standard deviation; CCS, Charlson comorbidity score.
References
- A new method of classifying prognostic comorbidity in longitudinal studies: development and validation.J chronic Dis. 1987; 40: 373-383
- How to measure comorbidity. A critical review of available methods.J Clin Epidemiol. 2003; 56: 221-229
- Why summary comorbidity measures such as the Charlson comorbidity index and Elixhauser score work.Med Care. 2015; 53: e65-e72
- A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data.Med Care. 2009; 47: 626-633
- A combined comorbidity score predicted mortality in elderly patients better than existing scores.J Clin Epidemiol. 2011; 64: 749-759
- A new comorbidity index: the health-related quality of life comorbidity index.J Clin Epidemiol. 2011; 64: 309-319
- A chronic disease score with empirically derived weights.Med Care. 1995; 33: 783-795
- Adapting the Rx-Risk-V for mortality prediction in outpatient populations.Med Care. 2006; 44: 793-797
- Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries.Am J Epidemiol. 2011; 173: 676-682
- Regression coefficients and scoring rules.J Clin Epidemiol. 1996; 49: 819
- Should scoring rules be based on odds ratios or regression coefficients?.J Clin Epidemiol. 2002; 55: 1054-1055
- Presentation of multivariate data for clinical use: the Framingham Study risk score functions.Stat Med. 2004; 23: 1631-1660
- Recent advances in the utility and use of the General Practice Research Database as an example of a UK Primary Care Data resource.Ther Adv Drug Saf. 2012; 3: 89-99
- Validity of diagnostic coding within the General Practice Research Database: a systematic review.Br J Gen Pract. 2010; 60: e128-e136
- Adaptation and validation of the Charlson Index for Read/OXMIS coded databases.BMC Fam Pract. 2010; 11: 1
- Improved comorbidity adjustment for predicting mortality in Medicare populations.Health Serv Res. 2003; 38: 1103-1120
- Assessing the performance of prediction models: a framework for traditional and novel measures.Epidemiology. 2010; 21: 128-138
- Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis.Springer, New York2001
- Applied logistic regression.2nd ed. Wiley, New York2000
- Coefficients of determination for multiple logistic regression analysis.Am Stat. 2000; 54: 17-24
- A note on a general definition of the coefficient of determination.Biometrika. 1991; 78: 691-692
- Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.Stat Med. 2008; 27 (discussion 207–12): 157-172
- Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures.Ann Intern Med. 2009; 150: 795-802
- Comorbidity risk-adjustment measures were developed and validated for studies of antibiotic-resistant infections.J Clin Epidemiol. 2006; 59: 1266-1273
- Relevance of the c-statistic when evaluating risk-adjustment models in surgery.J Am Coll Surg. 2012; 214: 822-830
- Use and misuse of the receiver operating characteristic curve in risk prediction.Circulation. 2007; 115: 928-935
Article info
Publication history
Published online: May 12, 2016
Accepted:
March 29,
2016
Footnotes
Funding: None.
Conflicts of interest: V.M. is an employee and hold stock and stock options in Merck & Co., Inc., a pharmaceutical company which manufactures numerous products. This research is not product related. C.J.G. is a retired employee of Merck & Co., Inc., a pharmaceutical manufacturer. This research is not product related. H.B.M., D.A., and M.L.J. declared no conflicts of interest.
Identification
Copyright
© 2016 Elsevier Inc. All rights reserved.
ScienceDirect
Access this article on ScienceDirectLinked Article
- Comment by M.E. Charlson and M. WellsJournal of Clinical EpidemiologyVol. 79
- PreviewThe Mehta et al.[1] study conducted an empirical evaluation of different methods of weighting the 17 chronic conditions in the original 1987 Charlson [2] comorbidity index. In summarizing the original study, the authors missed that the validation used 10-year mortality rates in the breast cancer population rather than one-year mortality rates in hospitalized patients. This is a critical difference, since any estimate of predictive power in one population will deteriorate in a subsequent population, particularly if it is overfit.
- Full-Text
- Preview
- Editorial comment: Ratios should be multiplied, not addedJournal of Clinical EpidemiologyVol. 79
- PreviewIt is a good thing that widely used instruments, including those whose validations have been published in peer reviewed journals, are regularly evaluated, as this ensures further development and improvement. At the same time, this can be challenging both for developers and users if an instrument as designed has been working already for a long time.
- Full-Text
- Preview
