Abstract
Objective
To evaluate limitations of common statistical modeling approaches in deriving clinical
prediction models and explore alternative strategies.
Study Design and Setting
A previously published model predicted the likelihood of having a mutation in germline
DNA mismatch repair genes at the time of diagnosis of colorectal cancer. This model
was based on a cohort where 38 mutations were found among 870 participants, with validation
in an independent cohort with 35 mutations. The modeling strategy included stepwise
selection of predictors from a pool of over 37 candidate predictors and dichotomization
of continuous predictors. We simulated this strategy in small subsets of a large contemporary
cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling
approaches. All models were evaluated according to bias and discriminative ability
(concordance index, c) in independent data.
Results
We found over 50% bias for five of six originally selected predictors, unstable model
specification, and poor performance at validation (median c = 0.74). A small validation sample hampered stable assessment of performance. Model
prespecification based on external knowledge and using continuous predictors led to
better performance (c = 0.836 and c = 0.852 with 38 and 2,051 events respectively).
Conclusion
Prediction models perform poorly if based on small numbers of events and developed
with common but suboptimal statistical approaches. Alternative modeling strategies
to best exploit available predictive information need wider implementation, with collaborative
research to increase sample sizes.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine.CA Cancer J Clin. 2016; 66: 370-374
- Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration.Ann Intern Med. 2015; 162: W1-W73
- A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes.J Clin Epidemiol. 2008; 61: 331-343
- Prognostic models: a methodological framework and review of models for breast cancer.Cancer Invest. 2009; 27: 235-243
- Reporting methods in studies developing prognostic models in cancer: a review.BMC Med. 2010; 8: 20
- Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting.BMC Med. 2011; 9: 103
- Reporting and methods in clinical prediction research: a systematic review.PLoS Med. 2012; 9: 1-12
- A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods.J Clin Epidemiol. 2013; 66: 268-277
- Clinical prediction models: a practical approach to development, validation, and updating.Springer, New York2009
- Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.J Clin Epidemiol. 2005; 58: 475-483
- Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.Stat Med. 2016; 35: 214-226
- Flawed external validation study of the ADNEX model to diagnose ovarian cancer.Gynecol Oncol Rep. 2016; 18: 49-50
- External validation of multivariable prediction models: a systematic review of methodological conduct and reporting.BMC Med Res Methodol. 2014; 14: 40
- External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination.J Clin Epidemiol. 2015; 68: 25-34
- The diagnostic value of scoring models for organic and non-organic gastrointestinal disease, including the irritable-bowel syndrome.Med Decis Making. 1994; 14: 208-216
- Identification and survival of carriers of mutations in DNA mismatch-repair genes in colon cancer.N Engl J Med. 2006; 354: 2751-2763
- Comparison of prediction models for Lynch syndrome among individuals with colorectal cancer.J Natl Cancer Inst. 2015; 108: 18
- Guidelines on genetic evaluation and management of Lynch syndrome: a consensus statement by the US Multi-Society Task Force on colorectal cancer.Gastroenterology. 2014; 147: 502-526
- ACG clinical guideline: genetic testing and management of hereditary gastrointestinal cancer syndromes.Am J Gastroenterol. 2015; 110: 223-262
- Prediction of MLH1 and MSH2 mutations in Lynch syndrome.JAMA. 2006; 296: 1469-1478
- The PREMM(1,2,6) model predicts risk of MLH1, MSH2, and MSH6 germline mutations based on cancer history.Gastroenterology. 2011; 140: 73-81
- Development and validation of the PREMM5 model for comprehensive risk assessment of Lynch syndrome.J Clin Oncol. 2017; 35: 2165-2172
Barnetson RA, Appendix, Available at http://www.nejm.org/doi/suppl/10.1056/NEJMoa053493/suppl_file/nejm_barnetson_2751sa1.pdf. 2006. Accessed May 1, 2017.
- Why most published research findings are false.PLoS Med. 2005; 2: e124
- Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis.J Clin Epidemiol. 1996; 49: 907-916
- Why most discovered true associations are inflated.Epidemiology. 2008; 19: 640-648
- Model uncertainty, data mining and statistical inference.J R Stat Soc Ser A. 1995; 158: 419-466
- Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis.J Clin Epidemiol. 1999; 52: 935-942
- Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets.Stat Med. 2000; 19: 1059-1079
- Dichotomizing continuous predictors in multiple regression: a bad idea.Stat Med. 2006; 25: 127-141
- Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model.Stat Med. 2016; 35: 4124-4135
- What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models.Psychosom Med. 2004; 66: 411-421
- Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints.BMC Med Res Methodol. 2014; 14: 137
- Con: most clinical risk scores are useless.Nephrol Dial Transplant. 2017; 32: 752-755
- Data reduction for prediction: robust coding of age and family history for the risk of having a genetic mutation.Stat Med. 2007; 26: 5545-5556
- Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis.Springer, New York2015
- Selection of important variables and determination of functional form for continuous predictors in multivariable model building.Stat Med. 2007; 26: 5512-5528
- Prediction models need appropriate internal, internal-external, and external validation.J Clin Epidemiol. 2016; 69: 245-247
- Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med. 1996; 15: 361-387
- Assessing the performance of prediction models: a framework for traditional and novel measures.Epidemiology. 2010; 21: 128-138
- A calibration hierarchy for risk models was defined: from utopia to empirical data.J Clin Epidemiol. 2016; 74: 167-176
- Towards better clinical prediction models: seven steps for development and an ABCD for validation.Eur Heart J. 2014; 35: 1925-1931
- Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events.Stat Med. 2016; 35: 1159-1177
- Performance of Firth-and logF-type penalized methods in risk prediction for small or sparse binary data.BMC Med Res Methodol. 2017; 17: 33
- Bootstrap investigation of the stability of a Cox regression model.Stat Med. 1989; 8: 771-783
- Backward, forward and stepwise automated subset selection algorithms: frequency of obtaining authentic and noise variables.Br J Math Stat Psychol. 1992; 45: 265-282
- Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality.J Clin Epidemiol. 2004; 57: 1138-1146
- The ASA's statement on p-values: context, process, and purpose.Am Stat. 2016; 70: 129-133
- Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations.Eur J Epidemiol. 2016; 31: 337-350
- Statistical inference in abstracts of major medical and epidemiology journals 1975-2014: a systematic review.Eur J Epidemiol. 2017; 32: 21-29
- Negative consequences of dichotomizing continuous predictor variables.J Marketing Res. 2003; 40: 366-371
- The cost of dichotomising continuous variables.BMJ. 2006; 332: 1080
- Analysis by categorizing or dichotomizing continuous variables is inadvisable: an example from the natural history of unruptured aneurysms.AJNR Am J Neuroradiol. 2011; 32: 437-440
- Dichotomizing continuous variables in statistical analysis.Med Decis Making. 2012; 32: 225-226
- Clinical utility of risk models to refer patients with adnexal masses to specialized oncology care: multicenter external validation using decision curve analysis.Clin Cancer Res. 2017; 23: 5082-5090
- Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.Med Decis Making. 2001; 21: 45-56
- On measuring and correcting the effects of data mining and model selection.J Am Stat Assoc. 1998; 93: 120-131
- The elements of statistical learning: data mining, inference, and prediction.Springer, New York2001
- Visualizing risk prediction models.PLoS One. 2015; 10: e0132614
- Predictors of 30-day mortality in the era of reperfusion for acute myocardial infarction. Results from an international trial of 41,021 patients. GUSTO-I Investigators.Circulation. 1995; 91: 1659-1668
- Internal and external validation of predictive models: a simulation study of bias and precision in small samples.J Clin Epidemiol. 2003; 56: 441-447
- Regression, prediction and shrinkage.J R Stat Soc Ser B. 1983; 45: 311-354
- Regression and shrinkage via the Lasso.J R Stat Soc Ser B. 1996; 58: 267-288
- Bayesian perspectives for epidemiological research. II. Regression analysis.Int J Epidemiol. 2007; 36: 195-202
- Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example.J Clin Epidemiol. 2004; 57: 1262-1270
- External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges.BMJ. 2016; 353: i3140
- A new framework to enhance the interpretation of external validation studies of clinical prediction models.J Clin Epidemiol. 2015; 68: 279-289
- A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis.Stat Med. 2013; 32: 3158-3180
- Prediction models for cardiovascular disease risk in the general population: systematic review.BMJ. 2016; 353: i2416
- Translating clinical research into clinical practice: impact of using prediction rules to make decisions.Ann Intern Med. 2006; 144: 201-209
- How to make more published research true.PLoS Med. 2014; 11: e1001747
- Assessment of the accuracy of diagnostic tests: the cross-sectional study.J Clin Epidemiol. 2003; 56: 1118-1128
- Calibration of risk prediction models: impact on decision-analytic performance.Med Decis Making. 2015; 35: 162-169
Article info
Publication history
Published online: November 23, 2017
Accepted:
November 17,
2017
Footnotes
Conflict of interest: The authors have declared that no conflicts of interest exist.
E.W.S. is supported by U01 NS086294 from the NIH and by grant 602150 (CENTER-TBI) from the European Union's FP7 Program.
Identification
Copyright
© 2017 Elsevier Inc. All rights reserved.
