Abstract
Objectives
Calibration is often thought to assess the bias of a clinical prediction rule. In
particular, if the rule is based on a linear logistic model, it is often assumed that
an overestimation of all coefficients results in a calibration slope less than 1 and
an underestimation in a slope larger than 1.
Study Design and Setting
We investigate the relation of the bias and the residual variation of clinical prediction
rules with the typical behavior of calibration plots and calibration slopes, using
some artificial examples.
Results
Calibration is not only sensitive to the bias of the clinical prediction rule but
also to the residual variation. In some circumstances, the effects may cancel out,
resulting in a misleading perfect calibration.
Conclusion
Poor calibration is a clear indication of limited usefulness of a clinical prediction
rule. However, a perfect calibration should be interpreted with care as this may happen
even for a biased prediction rule.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med. 1996; 15: 361-387
- The use of resampling methods to simplify regression models in medical statistics.J R Stat Soc Ser C Applied Stat. 1999; 48: 313-329
- Prognostic modeling with logistic regression analysis: In search of a sensible strategy in small data sets.Med Decis Making. 2001; 21: 45-56
- External validation is necessary in prediction research: a clinical example.J Clin Epidemiol. 2003; 56: 826-832
- Risk prediction models: II. External validation, model updating, and impact assessment.Heart. 2012; 98: 691-698
- Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings.Epidemiol Infect. 2008; 136: 1628-1637
- What do we mean by validating a prognostic model?.Stat Med. 2000; 19: 453-473
- Prognosis and prognostic research: validating a prognostic model.BMJ. 2009; 338: b605
- Assessing the generalizability of prognostic information.Ann Intern Med. 1999; 130: 515-524
- Criteria for scientific evaluation of novel markers: a perspective.Clin Chem. 2010; 56: 537-541
- Accuracy assessment of prediction in patient outcomes.J Eval Clin Pract. 2008; 14: 1-10
- Reporting and methods in clinical prediction research: a systematic review.PLoS Med. 2012; 9: e1001221
- Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.J Clin Epidemiol. 2005; 58: 475-483
- Validation of probabilistic predictions.Med Decis Making. 1993; 13: 49-58
- Two further applications of a model for binary regression.Biometrika. 1958; 45: 562-565
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology. 1982; 143: 29-36
- Updating methods improved the performance of a clinical prediction model in new patients.J Clin Epidemiol. 2008; 61: 76-86
- Robust locally weighted regression and smoothing scatterplots.J Am Stat Assoc. 1979; 74: 829-836
- External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients.Am J Epidemiol. 2010; 172: 971-980
- On the relation between the shrinkage effect and a shrinkage method.Comput Stat. 1997; 12: 279-292
- A new measure of prognostic separation in survival data.Stat Med. 2004; 23: 723-748
Article info
Publication history
Published online: September 10, 2013
Accepted:
June 2,
2013
Footnotes
Conflict of interest: This material has neither been published nor is under consideration for publication elsewhere. There is no external funding involved in this study, and there are no possible conflicts of interest.
Identification
Copyright
© 2013 Elsevier Inc. Published by Elsevier Inc. All rights reserved.
