This paper is only available as a PDF. To read, Please Download here.
Abstract
Objective
To illustrate in-depth validation of prediction models developed on multicenter data.
Methods
For each hospital in a multicenter registry, we evaluated predictive performance of a 30-day mortality prediction model for transcatheter aortic valve implantation (TAVI) using the Netherlands heart registration (NHR) dataset. We measured discrimination and calibration per hospital in a leave-center-out analysis (LCOA). Meta-analysis was used to calculate I2 values per performance metric from the LCOA and to compute mean and confidence interval (CI) estimates. Case-mix differences between studies were inspected using the framework of Debray et al. for understanding external validation. We also aimed to discover subgroups with high model prediction error and their distribution over the centers.
Results
We studied 16 hospitals with 11,599 TAVI patients with an early mortality of 3.7%. The models’ AUCs had a wide range between hospitals from 0.59 to 0.79 and miscalibration occurred in seven hospitals. Mean AUC from meta-analysis was 0.68 (95%CI 0.65-0.70). I 2 values were 0%, 74%, and 0% for AUC, calibration intercept and slope, respectively. Between-hospital case-mix differences were substantial and model transportability was low. One subgroup was discovered with marked global prediction error and was associated with poor performance on validation centers.
Conclusion
The illustrated combination of approaches provides useful insights to inspect multicenter-based prediction models, and exposes their limitations in transportability and performance variability when applied to different populations.
Graphical abstract
Graphical Abstract
Keywords
Article info
Publication history
Accepted:
February 14,
2023
Received in revised form:
January 30,
2023
Received:
September 22,
2022
Publication stage
In Press Journal Pre-ProofIdentification
Copyright
© 2023 The Author(s). Published by Elsevier Inc.
User license
Creative Commons Attribution (CC BY 4.0) | How you can reuse 
Elsevier's open access license policy
Creative Commons Attribution (CC BY 4.0)
Permitted
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
Elsevier's open access license policy
00037-9&id=fx1.jpg){kind=link}