Abstract
Objective
Adding risk factors to a prediction model often increases the area under the receiver
operating characteristic curve (AUC) only slightly, particularly when the AUC of the
model was already high. We investigated whether a risk factor that minimally improves
the AUC may nevertheless improve the predictive ability of the model, assessed by
integrated discrimination improvement (IDI).
Study Design and Setting
We simulated data sets with risk factors and event status for 100,000 hypothetical
individuals and created prediction models with AUCs between 0.50 and 0.95. We added
a single risk factor for which the effect was modeled as a certain odds ratio (OR
2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03).
Results
Across all AUC values of the baseline model, for a risk factor with the same OR, both
ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment
in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline
model was >0.90.
Conclusion
When the addition of a risk factor shows minimal improvement in AUC, predicted risks
generally show minimal changes too. Updating risk models with strong risk factors
may be informative for a subgroup of individuals, but not at the population level.
The AUC may not be as insensitive as is frequently argued.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology. 1982; 143: 29-36
- Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.Stat Med. 2008; 27 (discussion 207–212): 157-172
- Assessing the performance of prediction models: a framework for traditional and novel measures.Epidemiology. 2010; 21: 128-138
- A method of comparing the areas under receiver operating characteristic curves derived from the same cases.Radiology. 1983; 148: 839-843
- Use and misuse of the receiver operating characteristic curve in risk prediction.Circulation. 2007; 115: 928-935
- Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models.Stat Med. 2012; 31: 101-113
- Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer.J Natl Cancer Inst. 2008; 100: 978-979
- Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker.Am J Epidemiol. 2004; 159: 882-890
- The limitations of risk factors as prognostic tools.N Engl J Med. 2006; 355: 2615-2617
- Interpreting incremental value of markers added to risk prediction models.Am J Epidemiol. 2012; 176: 473-481
- Incremental value of rare genetic variants for the prediction of multifactorial diseases.Genome Med. 2013; 5: 76
- Further insight into the incremental value of new markers: the interpretation of performance measures and the importance of clinical context.Am J Epidemiol. 2012; 176: 482-487
- Predictive testing for complex diseases using multiple genes: fact or fiction?.Genet Med. 2006; 8: 395-400
- Constructing hypothetical risk data from the area under the ROC curve: modelling distributions of polygenic risk.PLoS One. 2016; 11: e0152359
- Evaluating the incremental value of new biomarkers with integrated discrimination improvement.Am J Epidemiol. 2011; 174: 364-374
- R: A Language and Environment for Statistical Computing.R Foundation for Statistical Computing, Vienna, Austria2015 (3.1.0 ed)
- Testing for improvement in prediction model performance.Stat Med. 2013; 32: 1467-1482
Article info
Publication history
Published online: July 15, 2016
Accepted:
July 8,
2016
Footnotes
Funding: This work was supported by a consolidator grant from the European Research Council (GENOMICMEDICINE).
Conflict of interest: None.
Identification
Copyright
© 2016 Elsevier Inc. All rights reserved.
