Original Article| Volume 66, ISSUE 11, P1296-1301, November 2013

Download started.


Calibration of clinical prediction rules does not just assess bias

  • Werner Vach
    Corresponding author. Tel.: +49 761 203 6722; fax: +49 761 203 6711.
    Clinical Epidemiology, Institute of Medical Biometry and Medical Informatics, Freiburg University Medical Center, Freiburg, Germany
    Search for articles by this author
Published:September 10, 2013DOI:



      Calibration is often thought to assess the bias of a clinical prediction rule. In particular, if the rule is based on a linear logistic model, it is often assumed that an overestimation of all coefficients results in a calibration slope less than 1 and an underestimation in a slope larger than 1.

      Study Design and Setting

      We investigate the relation of the bias and the residual variation of clinical prediction rules with the typical behavior of calibration plots and calibration slopes, using some artificial examples.


      Calibration is not only sensitive to the bias of the clinical prediction rule but also to the residual variation. In some circumstances, the effects may cancel out, resulting in a misleading perfect calibration.


      Poor calibration is a clear indication of limited usefulness of a clinical prediction rule. However, a perfect calibration should be interpreted with care as this may happen even for a biased prediction rule.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Harrell F.
        • Lee K.
        • Mark D.
        Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.
        Stat Med. 1996; 15: 361-387
        • Sauerbrei W.
        The use of resampling methods to simplify regression models in medical statistics.
        J R Stat Soc Ser C Applied Stat. 1999; 48: 313-329
        • Steyerberg E.
        • Eijkemans M.
        • Harrell F.
        • Habbema J.
        Prognostic modeling with logistic regression analysis: In search of a sensible strategy in small data sets.
        Med Decis Making. 2001; 21: 45-56
        • Bleeker S.
        • Moll H.
        • Steyerberg E.
        • Donders A.
        • Derksen-Lubsen G.
        • Grobbee D.
        • et al.
        External validation is necessary in prediction research: a clinical example.
        J Clin Epidemiol. 2003; 56: 826-832
        • Moons K.G.M.
        • Kengne A.P.
        • Grobbee D.E.
        • Royston P.
        • Vergouwe Y.
        • Altman D.G.
        • et al.
        Risk prediction models: II. External validation, model updating, and impact assessment.
        Heart. 2012; 98: 691-698
        • Schuetz P.
        • Koller M.
        • Christ-Crain M.
        • Steyerberg E.
        • Stolz D.
        • Mueller C.
        • et al.
        Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings.
        Epidemiol Infect. 2008; 136: 1628-1637
        • Altman D.
        • Royston P.
        What do we mean by validating a prognostic model?.
        Stat Med. 2000; 19: 453-473
        • Altman D.G.
        • Vergouwe Y.
        • Royston P.
        • Moons K.G.M.
        Prognosis and prognostic research: validating a prognostic model.
        BMJ. 2009; 338: b605
        • Justice A.
        • Covinsky K.
        • Berlin J.
        Assessing the generalizability of prognostic information.
        Ann Intern Med. 1999; 130: 515-524
        • Moons K.G.M.
        Criteria for scientific evaluation of novel markers: a perspective.
        Clin Chem. 2010; 56: 537-541
        • Bartfay E.
        • Bartfay W.J.
        Accuracy assessment of prediction in patient outcomes.
        J Eval Clin Pract. 2008; 14: 1-10
        • Bouwmeester W.
        • Zuithoff N.P.A.
        • Mallett S.
        • Geerlings M.I.
        • Vergouwe Y.
        • Steyerberg E.W.
        • et al.
        Reporting and methods in clinical prediction research: a systematic review.
        PLoS Med. 2012; 9: e1001221
        • Vergouwe Y.
        • Steyerberg E.
        • Eijkemans M.
        • Habbema J.
        Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.
        J Clin Epidemiol. 2005; 58: 475-483
        • Miller M.
        • Langefeld C.
        • Tierney W.
        • Hui S.
        • McDonald C.
        Validation of probabilistic predictions.
        Med Decis Making. 1993; 13: 49-58
        • Cox D.
        Two further applications of a model for binary regression.
        Biometrika. 1958; 45: 562-565
        • Hanley J.
        • McNeil B.
        The meaning and use of the area under a receiver operating characteristic (ROC) curve.
        Radiology. 1982; 143: 29-36
        • Janssen K.J.M.
        • Moons K.G.M.
        • Kalkman C.J.
        • Grobbee D.E.
        • Vergouwe Y.
        Updating methods improved the performance of a clinical prediction model in new patients.
        J Clin Epidemiol. 2008; 61: 76-86
        • Cleveland W.
        Robust locally weighted regression and smoothing scatterplots.
        J Am Stat Assoc. 1979; 74: 829-836
        • Vergouwe Y.
        • Moons K.G.M.
        • Steyerberg E.W.
        External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients.
        Am J Epidemiol. 2010; 172: 971-980
        • Vach W.
        On the relation between the shrinkage effect and a shrinkage method.
        Comput Stat. 1997; 12: 279-292
        • Royston P.
        • Sauerbrei W.
        A new measure of prognostic separation in survival data.
        Stat Med. 2004; 23: 723-748