Abstract
Background and objective
There is growing interest in developing prediction models. The accuracy of such models
when applied in new patient samples is commonly lower than estimated from the development
sample. This may be because of differences between the samples and/or because the
developed model was overfitted (too optimistic). Various methods, including bootstrapping
techniques exist for afterwards shrinking the regression coefficients and the model's
discrimination and calibration for overoptimism. Penalized maximum likelihood estimation
(PMLE) is a more rigorous method because adjustment for overfitting is directly built
into the model development, instead of relying on shrinkage afterwards. PMLE has been
described mainly in the statistical literature and is rarely applied to empirical
data. Using empirical data, we illustrate the use of PMLE to develop a prediction
model.
Methods
The accuracy of the final PMLE model will be contrasted with the final models derived
by ordinary stepwise logistic regression without and with shrinkage afterwards. The
potential advantages and disadvantages of PMLE over the other two strategies are discussed.
Results
PMLE leads to smaller prediction errors, provides for model reduction to a user-defined
degree, and may differently shrink each predictor for overoptimism without sacrificing
much discriminative accuracy of the model.
Conclusion
PMLE is an easily applicable and promising method to directly adjust clinical prediction
models for overoptimism.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Searching for clinical prediction rules in MEDLINE.J Am Med Inform Assoc. 2001; 8: 391-397
- Comparison of a clinical probability estimate and two clinical models in patients with suspected pulmonary embolism.Thromb Haemost. 2000; 83: 199-203
- External validation is necessary in prediction research: a clinical example.J Clin Epidemiol. 2003; 56: 826-832
- External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study.Intensive Care Med. 2003; 29: 249-256
- A cross-validation of risk-scores for coronary heart disease mortality based on data from the Glostrup Population Studies and Framingham Heart Study.Int J Epidemiol. 2002; 31: 817-822
- Prediction rules for complications in coronary bypass surgery: a comparison and methodological critique.Med Care. 2000; 38: 820-835
- Clinical prediction rules: a review and suggested modifications of methodological standards.JAMA. 1997; 277: 488-494
- Assessing the generalizability of prognostic information.Ann Intern Med. 1999; 130: 515-524
- Methodologic standards for the development of clinical decision rules in emergency medicine.Ann Emerg Med. 1999; 33: 437-447
- What do we mean by validating a prognostic model?.Stat Med. 2000; 19: 453-473
- Users' guides to the medical literature: XXII: How to use articles about clinical decision rules.JAMA. 2000; 284: 79-84
- Internal and external validation of predictive models: a simulation study of bias and precision in small samples.J Clin Epidemiol. 2003; 56: 441-447
- Construction, validation and updating of a prognostic model for kidney graft survival.Stat Med. 1995; 14: 1999-2008
- Validation, calibration, revision and combination of prognostic survival models.Stat Med. 2000; 19: 3401-3415
- Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med. 1996; 15: 361-387
- Regression modeling strategies.Springer-Verlag, New York2001
- An introduction to the bootstrap. Monographs on Statistics and Applied Probability.Chapman & Hall, New York1993
- Predictive value of statistical models.Stat Med. 1990; 9: 1303-1325
- Bootstrap investigation of the stability of a Cox regression model.Stat Med. 1989; 8: 771-783
- Model uncertainty, data mining, and statistical inference.J R Stat Soc Ser A. 1995; 158: 419-466
- Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis.J Clin Epidemiol. 1996; 49: 907-916
- Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis.J Clin Epidemiol. 1999; 52: 935-942
- Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.Med Decis Making. 2001; 21: 45-56
- Simplifying a prognostic model: a simulation study based on clinical data.Stat Med. 2002; 21: 3803-3822
- Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.J Clin Epidemiol. 2001; 54: 774-781
- Shrinkage and penalized likelihood methods to improve diagnostic accuracy.Stat Neerl. 2001; 55: 17-34
- Flexible methods for analysing survival data using splines, with applications to breast cancer prognosis.J Am Stat Assoc. 1992; 87: 942-951
- Penalized likelihood in Cox regression.Stat Med. 1994; 13: 2427-2436
- Development of a clinical prediction model for an ordinal outcome: the World Health Organization Multicentre Study of Clinical Signs and Etiological Agents of Pneumonia, Sepsis and Meningitis in Young Infants.Stat Med. 1998; 17: 909-944
- Applied regression analysis.3rd ed. Wiley, New York1998
- A note on the generalized information criterion for choice of a model.Biometrika. 1980; 67: 413-418
- A normal perfusion lung scan in patients with clinically suspected pulmonary embolism: frequency and clinical validity.Chest. 1995; 108: 170-173
- Diagnostic utility of ultrasonography of leg veins in patients suspected of having pulmonary embolism.Ann Intern Med. 1997; 126: 775-781
- Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves.Med Decis Making. 1997; 17: 447-454
- Redundancy of single diagnostic test evaluation.Epidemiology. 1999; 10: 276-281
- Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy.J Clin Epidemiol. 1995; 48: 1495-1501
- A simulation study of the number of events per variable in logistic regression analysis.J Clin Epidemiol. 1996; 49: 1373-1379
- Application of shrinkage techniques in logistic regression analysis: a case study.Stat Neerl. 2001; 55: 76-88
Article info
Publication history
Accepted:
January 20,
2004
Identification
Copyright
© 2004 Elsevier Inc. Published by Elsevier Inc. All rights reserved.
