Adding risk factors to a prediction model often increases the area under the receiver operating characteristic curve (AUC) only slightly, particularly when the AUC of the model was already high. We investigated whether a risk factor that minimally improves the AUC may nevertheless improve the predictive ability of the model, assessed by integrated discrimination improvement (IDI).
Study Design and Setting
We simulated data sets with risk factors and event status for 100,000 hypothetical individuals and created prediction models with AUCs between 0.50 and 0.95. We added a single risk factor for which the effect was modeled as a certain odds ratio (OR 2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03).
Across all AUC values of the baseline model, for a risk factor with the same OR, both ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline model was >0.90.
When the addition of a risk factor shows minimal improvement in AUC, predicted risks generally show minimal changes too. Updating risk models with strong risk factors may be informative for a subgroup of individuals, but not at the population level. The AUC may not be as insensitive as is frequently argued.
To read this article in full you will need to make a payment
Purchase one-time access:Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
One-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:Subscribe to Journal of Clinical Epidemiology
Already a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology. 1982; 143: 29-36
- Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.Stat Med. 2008; 27 (discussion 207–212): 157-172
- Assessing the performance of prediction models: a framework for traditional and novel measures.Epidemiology. 2010; 21: 128-138
- A method of comparing the areas under receiver operating characteristic curves derived from the same cases.Radiology. 1983; 148: 839-843
- Use and misuse of the receiver operating characteristic curve in risk prediction.Circulation. 2007; 115: 928-935
- Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models.Stat Med. 2012; 31: 101-113
- Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer.J Natl Cancer Inst. 2008; 100: 978-979
- Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker.Am J Epidemiol. 2004; 159: 882-890
- The limitations of risk factors as prognostic tools.N Engl J Med. 2006; 355: 2615-2617
- Interpreting incremental value of markers added to risk prediction models.Am J Epidemiol. 2012; 176: 473-481
- Incremental value of rare genetic variants for the prediction of multifactorial diseases.Genome Med. 2013; 5: 76
- Further insight into the incremental value of new markers: the interpretation of performance measures and the importance of clinical context.Am J Epidemiol. 2012; 176: 482-487
- Predictive testing for complex diseases using multiple genes: fact or fiction?.Genet Med. 2006; 8: 395-400
- Constructing hypothetical risk data from the area under the ROC curve: modelling distributions of polygenic risk.PLoS One. 2016; 11: e0152359
- Evaluating the incremental value of new biomarkers with integrated discrimination improvement.Am J Epidemiol. 2011; 174: 364-374
- R: A Language and Environment for Statistical Computing.R Foundation for Statistical Computing, Vienna, Austria2015 (3.1.0 ed)
- Testing for improvement in prediction model performance.Stat Med. 2013; 32: 1467-1482
Published online: July 15, 2016
Accepted: July 8, 2016
Funding: This work was supported by a consolidator grant from the European Research Council (GENOMICMEDICINE).
Conflict of interest: None.
© 2016 Elsevier Inc. All rights reserved.