Abstract
Objective
Simulation studies suggest that the ratio of the number of events to the number of
estimated parameters in a logistic regression model should be not less than 10 or
20 to 1 to achieve reliable effect estimates. Applications of propensity score approaches
for confounding control in practice, however, do often not consider these recommendations.
Study Design and Setting
We conducted extensive Monte Carlo and plasmode simulation studies to investigate
the impact of propensity score model overfitting on the performance in estimating
conditional and marginal odds ratios using different established propensity score
inference approaches. We assessed estimate accuracy and precision as well as associated
type I error and type II error rates in testing the null hypothesis of no exposure
effect.
Results
For all inference approaches considered, our simulation study revealed considerably
inflated standard errors of effect estimates when using overfitted propensity score
models. Overfitting did not considerably affect type I error rates for most inference
approaches. However, because of residual confounding, estimation performance and type
I error probabilities were unsatisfactory when using propensity score quintile adjustment.
Conclusion
Overfitting of propensity score models should be avoided to obtain reliable estimates
of treatment or exposure effects in individual studies.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- The central role of the propensity score in observational studies for causal effects.Biometrika. 1983; 70: 41-55
- Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies.Stat Med. 2015; 34: 3661-3679
- Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group.Stat Med. 1998; 17: 2265-2281
- Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders.Am J Epidemiol. 2003; 158: 3
- A simulation study of the number of events per variable in logistic regression analysis.J Clin Epidemiol. 1996; 49: 1373-1379
- Regression modeling strategies.Springer Science & Business Media, New York2001: 61
- Relaxing the rule of ten events per variable in logistic and Cox regression.Am J Epidemiol. 2007; 165: 710-718
- Matching using estimated propensity scores: relating theory to practice.Biometrics. 1996; 52: 249-264
- The use of propensity score matching in the evaluation of labour market policies.(Working Paper 4) Dept. for Work and Pensions, London, England2002
- Sensitivity of propensity score methods to the specifications.Econ Lett. 2008; 98: 309-319
- Variable selection for propensity score models.Am J Epidemiol. 2006; 163: 1149-1156
Clarke KA, Kenkel B, Rueda MR. Misspecification and the propensity score: the possibility of overadjustment, 2011. Available at https://www.rochester.edu/college/psc/clarke/MissProp.pdf. Accessed January 06, 2016.
- On the specification of propensity scores, with applications to the analysis of trade policies.J Bus Econ Stat. 2009; 27: 397-415
- Covariate selection in high-dimensional propensity score analyses of treatment effects in small samples.Am J Epidemiol. 2011; 173: 1404-1413
- Variable selection and raking in propensity scoring.Stat Med. 2007; 26: 1022-1033
- Splines for trend analysis and continuous confounder control.Epidemiology. 2011; 22: 874-875
- Bias associated with using the estimated propensity score as a regression covariate.Stat Med. 2014; 33: 74-87
- Flexible regression models with cubic splines.Stat Med. 1989; 8: 551-561
- Confounding and collapsibility in causal inference.Stat Sci. 1999; 14: 29-46
- R: a language and environment for statistical computing.R Foundation for Statistical Computing, Vienna, Austria2015 (Available at) (Accessed September 05, 2016)
- On the existence of maximum likelihood estimates in logistic regression models.Biometrika. 1984; 71: 1-10
- Partial separation in logistic discrimination.Journal of the Royal Statistical Society. Series B (Methodological). 1989; 51: 109-116
- Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases.Comput Stat Data Anal. 2014; 72: 219-226
Available at http://biostat.mc.vanderbilt.edu/DataSets. Accessed April 29, 2016.
- The effectiveness of right heart catheterization in the initial care of critically III patients.JAMA. 1996; 276: 889-897
- Invited commentary: positivity in practice.Am J Epidemiol. 2010; 171: 674-677
- Variance reduction in randomised trials by inverse probability weighting using the propensity score.Stat Med. 2014; 33: 721-737
- Logistic regression modeling and the number of events per variable: selection bias dominates.J Clin Epidemiol. 2011; 64: 1464-1465
Article info
Publication history
Published online: August 03, 2016
Accepted:
May 11,
2016
Footnotes
Funding: T.S. and W.K.L. were supported by the Canadian Network for Observational Drug Effect Studies (CNODES). CNODES is funded by a grant from Health Canada, the Drug Safety and Effectiveness Network (DSEN), and the Canadian Institutes of Health Research (CIHR) grant number 111845-1. T.S. was also supported by funding from the Royal Children's Hospital Foundation to the Melbourne Children's Trial Centre. Research at the Murdoch Childrens Research Institute is supported by the Victorian Government's Operational Infrastructure Support Program. R.W.P. is supported in part by a National Scholar (Chercheur-national) of the Fonds de Recherche du Quebec–Sante (FQR-S) and is a member of the Research Institute of the McGill University Health Center, which is supported by core funds from FQR-S.
Identification
Copyright
© 2016 Elsevier Inc. All rights reserved.
ScienceDirect
Access this article on ScienceDirectLinked Article
- Overfitting in propensity score model: a commentary on “propensity score model overfitting led to inflated variance of estimated odds ratios” by Schuster et al.Journal of Clinical EpidemiologyVol. 88
- PreviewFirst, we would like to congratulate Tibor Schuster, Wilfrid Kouokam Low, and Robert W. Platt for their recently published paper “Propensity score model overfitting led to inflated variance of estimated odds ratios” discussing the influence of the ratio of the number of exposed to the number of covariates included in the propensity score (PS) model on the performance of treatment effect estimation [1]. We fully agree with the authors that this point is an important issue poorly discussed in the literature, although the absence of misspecification of the PS model is one of the four assumptions on which a PS analysis is based [2].
- Full-Text
- Preview
