Abstract
Objectives
To evaluate the predictive ability of a mortality prediction model in subgroups of
intensive care unit (ICU) patients and test the validity for monitoring the outcome.
Study Design and Setting
A Japanese ICU database was used for the analyses. Adults admitted to an ICU between
April 1, 2019, and March 31, 2020, were included. Nine clinically relevant subgroups
were selected, and we evaluated the discrimination and calibration of the Japan Risk
of Death model, a recalibrated Acute Physiology and Chronic Health Evaluation III-j
model. Funnel plots and exponentially weighted moving average (EWMA) charts were used
to check its validity for monitoring in-hospital mortality. If the predictive performance
was poor, the model was recalibrated and model performance was reassessed.
Results
The study population comprised 14,513 patients across nine subgroups. The in-hospital
mortality rate ranged from 11.3% to 30.9%. The calibration was poor in most subgroups,
and the funnel plots and EWMA charts frequently revealed “out-of-control” signals
crossing the control limit of three standard deviations (SDs). The calibration improved
after recalibration, and the number of “out-of-control” signals decreased.
Conclusion
When monitoring the quality of care among subgroups of patients, testing the predictive
ability and recalibration of the risk model are needed.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Mortality prediction in intensive care units including premorbid functional status improved performance and internal validity.J Clin Epidemiol. 2022; 142: 230-241
- Preventing hospital-associated venous thromboembolism.(Available at)https://www.ahrq.gov/patient-safety/resources/vtguide/index.htmlDate accessed: July 6, 2022
- Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021.Intensive Care Med. 2021; 47: 1181-1247
- The improvement guide: a practical approach to enhancing organizational performance.2nd ed. Jossey-Bass, San Francisco, CA2009
- Risk-adjusted continuous outcome monitoring with an EWMA chart: could it have detected excess mortality among intensive care patients at Bundaberg Base Hospital?.Crit Care Resusc. 2010; 12: 36-41
- Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures.J Intensive Care. 2021; 9: 18
- The APACHE III prognostic system.Chest. 1991; 100: 1619-1636
- The Japanese Intensive care PAtient Database (JIPAD): a national intensive care unit registry in Japan.J Crit Care. 2020; 55: 86-94
- Handling over-dispersion of performance indicators.Qual Saf Health Care. 2005; 14: 347-351
- Management strategies to effect change in intensive care units: lessons from the world of business. Part I. Targeting quality improvement initiatives.Ann Am Thorac Soc. 2014; 11: 264-269
- Effect of performance improvement programs on compliance with sepsis bundles and mortality: a systematic review and meta-analysis of observational studies.PLoS One. 2015; 10: e0125827
- The surviving sepsis campaign bundles and outcome: results from the international multicentre prevalence study on sepsis (the IMPreSS study).Intensive Care Med. 2015; 41: 1620-1628
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology. 1982; 143: 29-36
- Verification of forecasts expressed in terms of probability.Mon Weather Rev. 1950; 78: 1-3
- Assessing the performance of prediction models: a framework for traditional and novel measures.Epidemiology. 2010; 21: 128-138
- The present use of quality indicators in the intensive care unit.Acta Anaesthesiol Scand. 2012; 56: 1078-1083
- Calibration: the Achilles heel of predictive analytics.BMC Med. 2019; 17: 230
- Funnel plots for comparing institutional performance.Stat Med. 2005; 24: 1185-1202
- SAPS 3--From evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission.Intensive Care Med. 2005; 31: 1345-1355
- A calibration hierarchy for risk models was defined: from utopia to empirical data.J Clin Epidemiol. 2016; 74: 167-176
- Conventional risk prediction models fail to accurately predict mortality risk among patients with coronavirus disease 2019 in intensive care units: a difficult time to assess clinical severity and quality of care.J Intensive Care. 2021; 9: 42
Article info
Publication history
Published online: December 09, 2022
Accepted:
December 7,
2022
Footnotes
Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Declaration of interests: HE, NI, and HM are affiliated with the Department of Healthcare Quality Assessment at the University of Tokyo. The department is a social collaboration department supported by National Clinical Database, Johnson & Johnson K.K., Nipro Corporation, and Intuitive Surgical Sàrl.
Conflict of interest: HE, NI, and HM are affiliated with the Department of Healthcare Quality Assessment at the University of Tokyo. The department is a social collaboration department supported by grants from the National Clinical Database, Johnson & Johnson K.K., Nipro Corporation, and Intuitive Surgical Sàrl. The other authors do not have any competing interests.
Author Contributions: Hideki Endo: Conceptualization, Formal analysis, Methodology, Visualization, and Writing - original draft. Shigehiko Uchino: Conceptualization, Methodology, and Writing - review & editing. Satoru Hashimoto: Conceptualization, Methodology, and Writing - review & editing. Nao Ichihara: Methodology and Writing - review & editing. Hiroaki Miyata: Supervision and Writing - review & editing.
Identification
Copyright
© 2022 Elsevier Inc. All rights reserved.
