Abstract
Background and objective
Prediction models tend to perform better on data on which the model was constructed
than on new data. This difference in performance is an indication of the optimism
in the apparent performance in the derivation set. For internal model validation,
bootstrapping methods are recommended to provide biascorrected estimates of model
performance. Results are often accepted without sufficient regard to the importance
of external validation. This report illustrates the limitations of internal validation
to determine generalizability of a diagnostic prediction model to future settings.
Methods
A prediction model for the presence of serious bacterial infections in children with
fever without source was derived and validated internally using bootstrap resampling
techniques. Subsequently, the model was validated externally.
Results
In the derivation set (n = 376), nine predictors were identified. The apparent area under the receiver operating
characteristic curve (95% confidence interval) of the model was 0.83 (0.78–0.87) and
0.76 (0.67–0.85) after bootstrap correction. In the validation set (n = 179) the performance was 0.57 (0.47–0.67).
Conclusion
For relatively small data sets, internal validation of prediction models by bootstrap
techniques may not be sufficient and indicative for the model's performance in future
patients. External validation is essential before implementing prediction models in
clinical practice.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Regression models for prognostic prediction: advantages, problems, and suggested solutions.Cancer Treat Rep. 1985; 69: 1071-1077
- Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med. 1996; 15: 361-387
- Estimating the error rate of a prediction rule: improvement on cross-validation.J Am Stat Assoc. 1983; 78: 316-331
- Predictive value of statistical models.Stat Med. 1990; 9: 1303-1325
- Assessing the generalizability of prognostic information.Ann Intern Med. 1999; 130: 515-524
- Clinical prediction rules. Applications and methodological standards.N Engl J Med. 1985; 313: 793-799
- Redundancy of single diagnostic test evaluation.Epidemiology. 1999; 10: 276-281
- Clinical prediction rules. A review and suggested modifications of methodological standards.JAMA. 1997; 277: 488-494
- Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group.JAMA. 2000; 284: 79-84
- Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis.J Clin Epidemiol. 1999; 52: 935-942
- Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets.Stat Med. 2000; 19: 1059-1079
- An introduction to the bootstrap. Monographs on statistics and applied probability.Chapman & Hall, New York1993
- Improvements on cross-validation: the.632+ bootstrap method.J Am Stat Assoc. 1997; 92: 548-560
- Data splitting.Am Stat. 1990; 44: 140-147
- The use of resampling methods to simplify regression models in medical statistics.Journal of the Royal Statistical Society Series C: Applied Statistics. 1999; 48: 313-329
- Internal validation of predictive models. Efficiency of some procedures for logistic regression analysis.J Clin Epidemiol. 2001; 54: 774-781
- Prediction rules: statistical reproducibility and clinical similarity.Med Decis Making. 1992; 12: 286-287
- Fever.Pediatr Rev. 1998; 19: 401-408
- Fever without source: management of children 3 to 36 months of age.Pediatr Infect Dis J. 1992; 11: 146-151
- A problem-oriented patient classification system for general pediatrics II.Tijdschr Kindergeneesk. 1996; 64: 99-104
- Predictors of occult pneumococcal bacteremia in young febrile children.Ann Emerg Med. 1998; 31: 679-687
- Outpatient management without antibiotics of fever in selected infants [discussion: N Engl J Med 1994;330:939–40].N Engl J Med. 1993; 329: 1437-1441
- Probability of bacterial infections in febrile infants less than three months of age: a meta-analysis.Pediatr Infect Dis J. 1992; 11: 257-264
- Predicting serious bacterial infection in young children with fever without apparent source.Acta Paediatr. 2001; 90: 1226-1232
- Prediction of bacterial meningitis in children with meningeal signs: reduction of lumbar punctures.Acta Paediatr. 2001; 90: 611-617
- Signs of meningeal irritation at the emergency department: how often bacterial meningitis?.Pediatr Emerg Care. 2001; 17: 161-164
- Assessing rater performance without a “gold standard” using consensus theory.Med Decis Making. 1997; 17: 71-79
- Predictive value of abnormal physical examination findings in ill-appearing and well-appearing febrile children.Pediatrics. 1985; 76: 167-171
- Observation, history, and physical examination in diagnosis of serious illnesses in febrile children less than or equal to 24 months.J Pediatr. 1987; 110: 26-30
- Should blood cultures be obtained in the evaluation of young febrile children without evident focus of bacterial infection? A decision analysis of diagnostic management strategies.Pediatrics. 1989; 84: 18-27
- Practice guideline for the management of infants and children 0 to 36 months of age with fever without source.Pediatrics. 1993; 92: 1-12
- Identification of infants unlikely to have serious bacterial infection although hospitalized for suspected sepsis.J Pediatr. 1985; 107: 855-860
- Predictors of bacteremia in febrile children 3 to 36 months of age.Pediatrics. 2000; 106: 977-982
- Markers of serious illness in infants under 6 months old presenting to a children's hospital.Arch Dis Child. 1990; 65: 750-756
- A predictive model to estimate the risk of serious bacterial infections in febrile infants.Eur J Pediatr. 1996; 155: 468-473
- Duration of fever and its relationship to bacteremia in febrile outpatients three to 36 months old. The Occult Bacteremia Study Group.Pediatr Emerg Care. 1997; 13: 317-319
- A critical look at methods for handling missing covariates in epidemiologic regression analyses.Am J Epidemiol. 1995; 142: 1255-1264
- Regression with missing X's: a review.J Am Stat Assoc. 1992; 87: 1227-1237
- A note on the general definition of the coefficient of determination.Biometrika. 1991; 78: 691-692
- Shrinkage and penalized likelihood as methods to improve predictive accuracy.Stat Neerl. 2001; 55: 17-34
- Application of shrinkage techniques in logistic regression analysis: a case study.Stat Neerl. 2001; 55: 76-88
- The measurement of performance in probabilistic diagnosis.Methods Inf Med. 1978; 17: 227-237
- When should clinical guidelines be updated?.BMJ. 2001; 323: 155-157
Article info
Publication history
Accepted:
November 21,
2002
Identification
Copyright
© 2003 Elsevier Inc. Published by Elsevier Inc. All rights reserved.
