Journal of Clinical Epidemiology
Volume 57, Issue 12 , Pages 1262-1270 , December 2004

Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example

  • K.G.M. Moons

      Affiliations

    • Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 80035, 3508 TA Utrecht, The Netherlands
    • Corresponding Author InformationCorresponding author. Tel.: +31 30 2509368; fax: +31 30 2505480.
    • ,
  • A. Rogier T. Donders

      Affiliations

    • Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 80035, 3508 TA Utrecht, The Netherlands
    • Center for Biostatistics, Utrecht University, Utrecht, The Netherlands
    • ,
  • E.W. Steyerberg

      Affiliations

    • Department of Public Health, Erasmus Medical Center Rotterdam, Rotterdam, The Netherlands
    • ,
  • F.E. Harrell

      Affiliations

    • Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA

,Accepted 20 January 2004.

References 

  1. Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001;8:391–397
  2. Sanson B, Lijmer JG, Mac Gillavry MR, Turkstra F, Prins MH, Buller HR, et al. Comparison of a clinical probability estimate and two clinical models in patients with suspected pulmonary embolism. Thromb Haemost. 2000;83:199–203
  3. Bleeker SE, Moll HA, Steyerberg EW, Donders ART, Derksen-Lubsen G, Grobbee DE, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;56:826–832
  4. Beck DH, Smith GB, Pappachan JV, Millar B. External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study. Intensive Care Med. 2003;29:249–256
  5. Thomsen TF, McGee D, Davidsen M, Jorgensen T. A cross-validation of risk-scores for coronary heart disease mortality based on data from the Glostrup Population Studies and Framingham Heart Study. Int J Epidemiol. 2002;31:817–822
  6. Fortescue EB, Kahn K, Bates DW. Prediction rules for complications in coronary bypass surgery: a comparison and methodological critique. Med Care. 2000;38:820–835
  7. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules: a review and suggested modifications of methodological standards. JAMA. 1997;277:488–494
  8. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130:515–524
  9. Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33:437–447
  10. Altman DG, Royston P. What do we mean by validating a prognostic model?. Stat Med. 2000;19:453–473
  11. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS, et al. Users' guides to the medical literature: XXII: How to use articles about clinical decision rules. JAMA. 2000;284:79–84
  12. Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KGM. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441–447
  13. van Houwelingen JC, Thorogood J. Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med. 1995;14:1999–2008
  14. van Houwelingen JC. Validation, calibration, revision and combination of prognostic survival models. Stat Med. 2000;19:3401–3415
  15. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387
  16. Harrell FE. Regression modeling strategies. New York: Springer-Verlag; 2001;
  17. Efron B, Tibshirani R. An introduction to the bootstrap. Monographs on Statistics and Applied Probability. New York: Chapman & Hall; 1993;
  18. van Houwelingen JC, Le Cessie S. Predictive value of statistical models. Stat Med. 1990;9:1303–1325
  19. Altman DG, Andersen PK. Bootstrap investigation of the stability of a Cox regression model. Stat Med. 1989;8:771–783
  20. Chatfield C. Model uncertainty, data mining, and statistical inference. J R Stat Soc Ser A. 1995;158:419–466
  21. Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol. 1996;49:907–916
  22. Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999;52:935–942
  23. Steyerberg EW, Eijkemans MJ, Harrell FE, Habbema JD. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making. 2001;21:45–56
  24. Ambler G, Brady AR, Royston P. Simplifying a prognostic model: a simulation study based on clinical data. Stat Med. 2002;21:3803–3822
  25. Steyerberg EW, Harrell FE, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–781
  26. Houwelingen van JC. Shrinkage and penalized likelihood methods to improve diagnostic accuracy. Stat Neerl. 2001;55:17–34
  27. Gray RJ. Flexible methods for analysing survival data using splines, with applications to breast cancer prognosis. J Am Stat Assoc. 1992;87:942–951
  28. Verweij PJ, Van Houwelingen HC. Penalized likelihood in Cox regression. Stat Med. 1994;13:2427–2436
  29. Harrell FE, Margolis PA, Gove S, Mason KE, Mulholland EK, Lehmann D, et al. Development of a clinical prediction model for an ordinal outcome: the World Health Organization Multicentre Study of Clinical Signs and Etiological Agents of Pneumonia, Sepsis and Meningitis in Young Infants. Stat Med. 1998;17:909–944
  30. Draper NR, Smith H. Applied regression analysis. 3rd ed.. New York: Wiley; 1998;
  31. Atkinson AC. A note on the generalized information criterion for choice of a model. Biometrika. 1980;67:413–418
  32. van Beek EJR, Kuyer PMM, Schenk BE, Brandjes DPM, ten Cate JW, Buller HR. A normal perfusion lung scan in patients with clinically suspected pulmonary embolism: frequency and clinical validity. Chest. 1995;108:170–173
  33. Turkstra F, Kuijer PMM, van Beek EJR, Brandjes DPM, ten Cate JW, Buller HR. Diagnostic utility of ultrasonography of leg veins in patients suspected of having pulmonary embolism. Ann Intern Med. 1997;126:775–781
  34. Moons KG, Stijnen T, Michel BC, Buller HR, Van Es GA, Grobbee DE, et al. Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves. Med Decis Making. 1997;17:447–454
  35. Moons KG, van Es GA, Michel BC, Buller HR, Habbema JD, Grobbee DE. Redundancy of single diagnostic test evaluation. Epidemiology. 1999;10:276–281
  36. Concato J, Peduzzi P, Holford TR, Feinstein AR. Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy. J Clin Epidemiol. 1995;48:1495–1501
  37. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49:1373–1379
  38. Steyerberg EW, Eijkemans MJC, Habbema JDF. Application of shrinkage techniques in logistic regression analysis: a case study. Stat Neerl. 2001;55:76–88

PII: S0895-4356(04)00171-4

doi: 10.1016/j.jclinepi.2004.01.020

Journal of Clinical Epidemiology
Volume 57, Issue 12 , Pages 1262-1270 , December 2004

Article Tools

* Image Usage & Resolution