Advertisement

A framework for evaluating predictive models

      Abstract

      Predictive models provide estimates on an individual's probability of having a disease or developing a disease/disease outcome. Clinicians often use them to support clinical decision-making. Many prediction models are published annually; online versions of models (such as MDCalc and QxMD) facilitate their use at the point of care. However, before using a model, the clinician should first establish that the model has undergone external validation demonstrating satisfactory predictive performance. Ideally, the model should also demonstrate improved outcomes from an impact analysis. This article summarizes the basic steps of predictive model evaluation, and is followed by an application example.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Bonnett L.J.
        • Snell K.I.E.
        • Collins G.S.
        • Riley R.D.
        Guide to presenting clinical prediction models for use in clinical settings.
        BMJ. 2019; 365: I737https://doi.org/10.1136/bmj.l737
        • Hendriksen J.M.T.
        • Geersing G.J.
        • Moons K.G.M.
        • de Groot J.A.H.
        Diagnostic and prognostic prediction models.
        J Thromb Haemost. 2014; 11: 129-141
        • Alba A.C.
        • Agoritsas T.
        • Walsh M.
        • Hanna S.
        • Iorio A.
        • Devereaux P.J.
        • et al.
        Discrimination and calibration of clinical prediction models: users’ guides to the medical literature.
        JAMA. 2017; 318: 1377-1384
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • Gerds T.
        • Gonen M.
        • Obuchowski N.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Van Calster B.
        • Nieboer D.
        • Vergouwe Y.
        • De Cock B.
        • Pencina M.J.
        • Steyerberg E.W.
        A calibration hierarchy for risk models was defined: from utopia to empirical data.
        J Clin Epidemiol. 2016; 74: 167-176
        • Van Calster B.
        • McLernon D.J.
        • Van Smeden M.
        • Wynants L.
        • Steyerberg E.W.
        Calibration: the Achilles heel of predictive analytics.
        BMC Med. 2019; 17: 230
        • Steyerberg E.W.
        Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.
        Springer, New York2009
        • Toll D.B.
        • Janssen K.J.
        • Vergouwe Y.
        • Moons K.G.
        Validation, updating and impact of clinical prediction rules: a review.
        J Clin Epidemiol. 2008; 61: 1085-1094
        • Bleeker S.E.
        • Moll H.A.
        • Steyerberg E.W.
        • Donders A.R.
        • Derksen-Lubsen G.
        • Grobbee D.E.
        • et al.
        External validation is necessary in prediction research: a clinical example.
        J Clin Epidemiol. 2003; 56: 826-832
        • Wallace E.
        • Smith S.M.
        • Perera-Salazar R.
        • Vaucher P.
        • McCowan C.
        • Collins G.
        • et al.
        International diagnostic and Prognosis prediction (IDAPP) group. Framework for the impact analysis and implementation of clinical prediction rules (CPRs).
        BMC Med Inform Decis Mak. 2011; 11: 62
        • Lim W.S.
        • van der Eerden M.M.
        • Laing R.
        • Boersma W.G.
        • Karalus N.
        • Town G.I.
        • et al.
        Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study.
        Thorax. 2003; 58: 377-382
        • Capelastegui A.
        • España P.P.
        • Quintana J.M.
        • Areitio I.
        • Gorordo I.
        • Egurrola M.
        • et al.
        Validation of a predictive rule for the management of community-acquired pneumonia.
        Eur Respir J. 2006; 27: 151-157
        • Chalmers J.D.
        • Singanayagam A.
        • Akram A.R.
        • Mandal P.
        • Short P.M.
        • Choudhury G.
        • et al.
        Severity assessment tools for predicting mortality in hospitalised patients with community-acquired pneumonia. Systematic review and meta-analysis.
        Thorax. 2010; 65: 878-883

      Further readings

        • Steyerberg E.W.
        Clinical prediction models: a practical approach to development, validation, and updating.
        Springer, New York, NY2009 ([this book uses case studies to elegantly describe strategies for the development as well as evaluation of prediction models. Common pitfalls that lead to suboptimal models are addressed])
        • Alba A.C.
        • Agoritsas T.
        • Walsh M.
        • Hanna S.
        • Iorio A.
        • Devereaux P.J.
        • et al.
        Discrimination and calibration of clinical prediction models: users’ guides to the medical literature.
        JAMA. 2017; 318 ([provides a good illustration of using different classification cutoffs / decision thresholds for model selection, depending on clinical context]): 1377-1384
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • Gerds T.
        • Gonen M.
        • Obuchowski N.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21 ([this paper looks beyond statistical measures of discrimination and calibration, discusses reclassification metrics, and clarifies roles of decision-analytic measures such as decision curves in clinical decision-making]): 128-138
        • Wallace E.
        • Smith S.M.
        • Perera-Salazar R.
        • Vaucher P.
        • McCowan C.
        • Collins G.
        • et al.
        • International diagnostic and Prognosis prediction (IDAPP) group
        Framework for the impact analysis and implementation of clinical prediction rules (CPRs).
        BMC Med Inform Decis Mak. 2011; 11 ([challenges in designing and conducting impact analysis are described, with ways to overcome these]): 62
        • Van Calster B.
        • Vergouwe Y.
        • De Cock B.
        • Pencina M.J.
        • Steyerberg E.W.
        • et al.
        A alibration hierarchy for risk models was defined: from utopia to empirical data.
        J Clin Epidemiol. 2016; 74 ([a calibration hierarchy of risk models is defined, and their clinical utility in terms of decision curve analysis assessed. In so, the merits of models that achieve moderate calibration are discussed]): 167-176