Original Article| Volume 57, ISSUE 12, P1262-1270, December 2004

Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example


      Background and objective

      There is growing interest in developing prediction models. The accuracy of such models when applied in new patient samples is commonly lower than estimated from the development sample. This may be because of differences between the samples and/or because the developed model was overfitted (too optimistic). Various methods, including bootstrapping techniques exist for afterwards shrinking the regression coefficients and the model's discrimination and calibration for overoptimism. Penalized maximum likelihood estimation (PMLE) is a more rigorous method because adjustment for overfitting is directly built into the model development, instead of relying on shrinkage afterwards. PMLE has been described mainly in the statistical literature and is rarely applied to empirical data. Using empirical data, we illustrate the use of PMLE to develop a prediction model.


      The accuracy of the final PMLE model will be contrasted with the final models derived by ordinary stepwise logistic regression without and with shrinkage afterwards. The potential advantages and disadvantages of PMLE over the other two strategies are discussed.


      PMLE leads to smaller prediction errors, provides for model reduction to a user-defined degree, and may differently shrink each predictor for overoptimism without sacrificing much discriminative accuracy of the model.


      PMLE is an easily applicable and promising method to directly adjust clinical prediction models for overoptimism.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ingui B.J.
        • Rogers M.A.
        Searching for clinical prediction rules in MEDLINE.
        J Am Med Inform Assoc. 2001; 8: 391-397
        • Sanson B.
        • Lijmer J.G.
        • Mac Gillavry M.R.
        • Turkstra F.
        • Prins M.H.
        • Buller H.R.
        • ANTELOPE Study Group
        Comparison of a clinical probability estimate and two clinical models in patients with suspected pulmonary embolism.
        Thromb Haemost. 2000; 83: 199-203
        • Bleeker S.E.
        • Moll H.A.
        • Steyerberg E.W.
        • Donders A.R.T.
        • Derksen-Lubsen G.
        • Grobbee D.E.
        • Moons K.G.M.
        External validation is necessary in prediction research: a clinical example.
        J Clin Epidemiol. 2003; 56: 826-832
        • Beck D.H.
        • Smith G.B.
        • Pappachan J.V.
        • Millar B.
        External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study.
        Intensive Care Med. 2003; 29: 249-256
        • Thomsen T.F.
        • McGee D.
        • Davidsen M.
        • Jorgensen T.
        A cross-validation of risk-scores for coronary heart disease mortality based on data from the Glostrup Population Studies and Framingham Heart Study.
        Int J Epidemiol. 2002; 31: 817-822
        • Fortescue E.B.
        • Kahn K.
        • Bates D.W.
        Prediction rules for complications in coronary bypass surgery: a comparison and methodological critique.
        Med Care. 2000; 38: 820-835
        • Laupacis A.
        • Sekar N.
        • Stiell I.G.
        Clinical prediction rules: a review and suggested modifications of methodological standards.
        JAMA. 1997; 277: 488-494
        • Justice A.C.
        • Covinsky K.E.
        • Berlin J.A.
        Assessing the generalizability of prognostic information.
        Ann Intern Med. 1999; 130: 515-524
        • Stiell I.G.
        • Wells G.A.
        Methodologic standards for the development of clinical decision rules in emergency medicine.
        Ann Emerg Med. 1999; 33: 437-447
        • Altman D.G.
        • Royston P.
        What do we mean by validating a prognostic model?.
        Stat Med. 2000; 19: 453-473
        • McGinn T.G.
        • Guyatt G.H.
        • Wyer P.C.
        • Naylor C.D.
        • Stiell I.G.
        • Richardson W.S.
        • Evidence-Based Medicine Working Group
        Users' guides to the medical literature: XXII: How to use articles about clinical decision rules.
        JAMA. 2000; 284: 79-84
        • Steyerberg E.W.
        • Bleeker S.E.
        • Moll H.A.
        • Grobbee D.E.
        • Moons K.G.M.
        Internal and external validation of predictive models: a simulation study of bias and precision in small samples.
        J Clin Epidemiol. 2003; 56: 441-447
        • van Houwelingen J.C.
        • Thorogood J.
        Construction, validation and updating of a prognostic model for kidney graft survival.
        Stat Med. 1995; 14: 1999-2008
        • van Houwelingen J.C.
        Validation, calibration, revision and combination of prognostic survival models.
        Stat Med. 2000; 19: 3401-3415
        • Harrell Jr., F.E.
        • Lee K.L.
        • Mark D.B.
        Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.
        Stat Med. 1996; 15: 361-387
        • Harrell Jr., F.E.
        Regression modeling strategies.
        Springer-Verlag, New York2001
        • Efron B.
        • Tibshirani R.
        An introduction to the bootstrap. Monographs on Statistics and Applied Probability.
        Chapman & Hall, New York1993
        • van Houwelingen J.C.
        • Le Cessie S.
        Predictive value of statistical models.
        Stat Med. 1990; 9: 1303-1325
        • Altman D.G.
        • Andersen P.K.
        Bootstrap investigation of the stability of a Cox regression model.
        Stat Med. 1989; 8: 771-783
        • Chatfield C.
        Model uncertainty, data mining, and statistical inference.
        J R Stat Soc Ser A. 1995; 158: 419-466
        • Sun G.W.
        • Shook T.L.
        • Kay G.L.
        Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis.
        J Clin Epidemiol. 1996; 49: 907-916
        • Steyerberg E.W.
        • Eijkemans M.J.
        • Habbema J.D.
        Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis.
        J Clin Epidemiol. 1999; 52: 935-942
        • Steyerberg E.W.
        • Eijkemans M.J.
        • Harrell F.E.
        • Habbema J.D.
        Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.
        Med Decis Making. 2001; 21: 45-56
        • Ambler G.
        • Brady A.R.
        • Royston P.
        Simplifying a prognostic model: a simulation study based on clinical data.
        Stat Med. 2002; 21: 3803-3822
        • Steyerberg E.W.
        • Harrell Jr., F.E.
        • Borsboom G.J.
        • Eijkemans M.J.
        • Vergouwe Y.
        • Habbema J.D.
        Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.
        J Clin Epidemiol. 2001; 54: 774-781
        • Houwelingen van J.C.
        Shrinkage and penalized likelihood methods to improve diagnostic accuracy.
        Stat Neerl. 2001; 55: 17-34
        • Gray R.J.
        Flexible methods for analysing survival data using splines, with applications to breast cancer prognosis.
        J Am Stat Assoc. 1992; 87: 942-951
        • Verweij P.J.
        • Van Houwelingen H.C.
        Penalized likelihood in Cox regression.
        Stat Med. 1994; 13: 2427-2436
        • Harrell Jr., F.E.
        • Margolis P.A.
        • Gove S.
        • Mason K.E.
        • Mulholland E.K.
        • Lehmann D.
        • Muhe L.
        • Gatchalian S.
        • Eichenwald H.F.
        • WHO/ARI Young Infant Multicentre Study Group
        Development of a clinical prediction model for an ordinal outcome: the World Health Organization Multicentre Study of Clinical Signs and Etiological Agents of Pneumonia, Sepsis and Meningitis in Young Infants.
        Stat Med. 1998; 17: 909-944
        • Draper N.R.
        • Smith H.
        Applied regression analysis.
        3rd ed. Wiley, New York1998
        • Atkinson A.C.
        A note on the generalized information criterion for choice of a model.
        Biometrika. 1980; 67: 413-418
        • van Beek E.J.R.
        • Kuyer P.M.M.
        • Schenk B.E.
        • Brandjes D.P.M.
        • ten Cate J.W.
        • Buller H.R.
        A normal perfusion lung scan in patients with clinically suspected pulmonary embolism: frequency and clinical validity.
        Chest. 1995; 108: 170-173
        • Turkstra F.
        • Kuijer P.M.M.
        • van Beek E.J.R.
        • Brandjes D.P.M.
        • ten Cate J.W.
        • Buller H.R.
        Diagnostic utility of ultrasonography of leg veins in patients suspected of having pulmonary embolism.
        Ann Intern Med. 1997; 126: 775-781
        • Moons K.G.
        • Stijnen T.
        • Michel B.C.
        • Buller H.R.
        • Van Es G.A.
        • Grobbee D.E.
        • Habbema J.D.
        Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves.
        Med Decis Making. 1997; 17: 447-454
        • Moons K.G.
        • van Es G.A.
        • Michel B.C.
        • Buller H.R.
        • Habbema J.D.
        • Grobbee D.E.
        Redundancy of single diagnostic test evaluation.
        Epidemiology. 1999; 10: 276-281
        • Concato J.
        • Peduzzi P.
        • Holford T.R.
        • Feinstein A.R.
        Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy.
        J Clin Epidemiol. 1995; 48: 1495-1501
        • Peduzzi P.
        • Concato J.
        • Kemper E.
        • Holford T.R.
        • Feinstein A.R.
        A simulation study of the number of events per variable in logistic regression analysis.
        J Clin Epidemiol. 1996; 49: 1373-1379
        • Steyerberg E.W.
        • Eijkemans M.J.C.
        • Habbema J.D.F.
        Application of shrinkage techniques in logistic regression analysis: a case study.
        Stat Neerl. 2001; 55: 76-88