Abstract
Prognostic models are increasingly common in the biomedical literature. These models
are frequently evaluated with respect to their ability to discriminate between those
with and without an outcome. The area under the receiver-operating curve (AROC) is
often used to assess discrimination. In this study, we introduce a bootstrap method,
and, using Monte Carlo simulation, we compare three different bootstrap approaches
with four commonly used methods in their ability to accurately estimate 95% confidence
intervals (CIs) around the AROC for a simple prognostic model. We also evaluated the
power of a bootstrap method and the commonly used trapezoid rule to compare different
prognostic models. We show that several good methods exist for calculating 95% CIs
of AROC, but the maximum likelihood estimation method should not be used with small
sample sizes. We further show that for our simple prognostic model a bootstrap z-statistic
approach is preferred over the trapezoidal method when comparing the AROCs of two
related models.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Comparing tumour staging and grading systems.Stat Med. 2000; 19: 1997-2014
- Multivariate prognostic models.Stat Med. 1996; 15: 361-387
- Predicting clinical states in individual patients.Ann Intern Med. 1996; 125: 406-412
- Probabilistic prediction in patient management.Stat Med. 1986; 5: 421-433
- Predictive value of statistical models.Stat Med. 1990; 9: 1303-1325
- Goodness of fit tests for multiple logistic regression model.Commun Statist -Part A Theor Math. 1980; A9: 1043-1069
- Applied logistic regression. 2 ed. Wiley;, New York2000
- Prognostic modelling with logistic regression analysis.Stat Med. 2000; 19: 1059-1079
- Measuring the accuracy of diagnostic systems.Science. 1988; 240: 1285-1293
- ROC analysis applied to the evaluation of medical imaging techniques.Invest Radiol. 1979; 14: 109-121
- A method of comparing the areas under the receiver operating characteristic curves derived form the same cases.Radiol. 1983; 148: 839-843
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiol. 1982; 143: 29-36
- Verification bias in pediatric studies evaluating diagnostic tests.J Pediatr. 1993; 122: 585-590
- Signal detection theory and ROC analysis in psychology and diagnostics. 1st ed. Lawrence Erlbaum;, Mahwah, NJ1996
- A visicalc program for estimating the area under a receivor operating characteristic (ROC) curve.Med Decis Making. 1985; 5: 139-148
- The area above the ordinal dominance graph and the area below the receivor operating characteristic graph.J Math Psychol. 1975; 12: 387-415
- Comparing the areas under two or more correlated receiver operating characteristic curves.Biometrics. 1988; 44: 837-845
- Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets.Med Decis Making. 1998; 18: 110-121
- Confidence bands for receivor operating characteristic curves.Med Decis Making. 1993; 13: 191-197
- Maxium-liklihood estimation of parameters of signal detection theory and determination of confidence intervals-rating methods data.J Am Coll Cardiol. 1969; 6: 487-496
- Bootstrap methods. Wiley-Interscience;, New York1999
- Bootstrap methods and their applications. Cambridge University Press;, Cambridge2000
- A leisurely look at the bootstrap, the jackknife, and cross-validation.Am Statistician. 1983; 37: 36-48
- Bootstrap.Stat Sci. 1994; 9: 382-415
- Bootstrapping. Sage;, Newbury Park1993
- Which venous leg ulcers will heal with a limb compression bandage?.Am J Med. 2000; 109: 15-19
- Commentary.ACP Journal Club. 2001; 2: 77
Mooney, CZ. Monte Carlo simulation. Lewis-Beck, M. S. Thousand Oaks, CA: Sage; 1997. Quantitative applications in the social sciences.
- Simulation. Wiley-Interscience;, New York2000
- Venous leg ulcers.Arch Dermatol. 1999; 135: 920-926
- Definitions and guideline for assessment of wounds and evaluation of healing.Arch Dermatol. 1994; 130: 489-493
- Reliability and validity of the clinical interpretation of a healed chronic wound.Wound Rep Reg. 1996; 4: 335-338
Stata Statistical Software: Release 6.0. College Station, TX: Stata Corporation; 1999.
- Bootstrap confidence intervals:when, which, what? A practical guide for medical statisticians.Stat Med. 2000; 19: 1141-1164
- Graphical methods for data analysis. Wadsworth;, Boston1983
- Validation of a melanoma prognostic model.Arch Dermatol. 1998; 134: 1597-1601
- The changing prognosis of melanoma.Curr Oncol Rep. 2000; 2: 322-328
- Clinical correlates of Breslow thickness of malignant melanoma.Br J Dermatol. 2001; 144: 476-483
- Biased bootstrap methods for reducing the effects of contamination.J R Stat Soc. 1999; 61: 661-680
- Assessing the generalizability of prognostic information.Ann Intern Med. 1999; 130: 515-524
- Internal validation of predictive models.J Clin Epidemiol. 2001; 54: 774-781
- Prognostic modelling with logistic regression analysis.Stat Med. 2000; 19: 1059-1079
Article info
Publication history
Accepted:
November 15,
2001
Received in revised form:
August 22,
2001
Received:
October 26,
2000
Identification
Copyright
© 2002 Elsevier Science Inc. Published by Elsevier Inc. All rights reserved.
