Abstract
Objectives
The objective of the study is to present the Grading of Recommendations Assessment,
Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty
of evidence from modeling studies (i.e., certainty associated with model outputs).
Study Design and Setting
Expert consultations and an international multidisciplinary workshop informed development
of a conceptual approach to assessing the certainty of evidence from models within
the context of systematic reviews, health technology assessments, and health care
decisions. The discussions also clarified selected concepts and terminology used in
the GRADE approach and by the modeling community. Feedback from experts in a broad
range of modeling and health care disciplines addressed the content validity of the
approach.
Results
Workshop participants agreed that the domains determining the certainty of evidence
previously identified in the GRADE approach (risk of bias, indirectness, inconsistency,
imprecision, reporting bias, magnitude of an effect, dose–response relation, and the
direction of residual confounding) also apply when assessing the certainty of evidence
from models. The assessment depends on the nature of model inputs and the model itself
and on whether one is evaluating evidence from a single model or multiple models.
We propose a framework for selecting the best available evidence from models: 1) developing
de novo, a model specific to the situation of interest, 2) identifying an existing
model, the outputs of which provide the highest certainty evidence for the situation
of interest, either “off-the-shelf” or after adaptation, and 3) using outputs from
multiple models. We also present a summary of preferred terminology to facilitate
communication among modeling and health care disciplines.
Conclusion
This conceptual GRADE approach provides a framework for using evidence from models
in health decision-making and the assessment of certainty of evidence from a model
or models. The GRADE Working Group and the modeling community are currently developing
the detailed methods and related guidance for assessing specific domains determining
the certainty of evidence from models across health care–related disciplines (e.g.,
therapeutic decision-making, toxicology, environmental health, and health economics).
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- The role of quantitative models in science.in: Canham C.D. Cole J.J. Lauenroth W.K. Models in ecosystem science. Princeton University Press, Princeton, NJ2003: 13-31
- Models in science.in: Zalta E.N. The Stanford Encyclopedia of Philosophy. Spring 2017 Edition, Stanford, CA2017
- What is "quality of evidence" and why is it important to clinicians?.BMJ. 2008; 336: 995-998
- Evaluation (not validation) of quantitative models.Environ Health Perspect. 1998; 106: 1453-1460
- Model parameter estimation and uncertainty: a report of the ISPOR-SMDM modeling good research practices task force--6.Value Health. 2012; 15: 835-842
- Modeling good research practices--overview: a report of the ISPOR-SMDM modeling good research practices task force-1.Med Decis Making. 2012; 32: 667-677
- Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.Value Health. 2014; 17: 174-182
- Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-7.Med Decis Making. 2012; 32: 733-743
- Modeling using discrete event simulation: a report of the ISPOR-SMDM modeling good research practices task force-4.Med Decis Making. 2012; 32: 701-711
- Selecting a dynamic simulation modeling method for health care delivery research-part 2: report of the ISPOR Dynamic Simulation Modeling Emerging Good Practices Task Force.Value Health. 2015; 18: 147-160
- Applying dynamic simulation modeling methods in health care delivery research-the SIMULATE checklist: report of the ISPOR simulation modeling emerging good practices task force.Value Health. 2015; 18: 5-16
- Dynamic transmission modeling: a report of the ISPOR-SMDM modeling good research practices task force working group-5.Med Decis Making. 2012; 32: 712-721
- Conceptualizing a model: a report of the ISPOR-SMDM modeling good research practices task force-2.Med Decis Making. 2012; 32: 678-689
- State-transition modeling: a report of the ISPOR-SMDM modeling good research practices task force-3.Med Decis Making. 2012; 32: 690-700
- Improving model validation in health technology assessment: comments on guidelines of the ISPOR-SMDM modeling good research practices task force.Value Health. 2013; 16: 1106-1107
- Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--Modeling Studies.Value Health. 2003; 6: 9-17
- Reporting guidelines for modelling studies.BMC Med Res Methodol. 2012; 12: 168
- A systematic review of research guidelines in decision-analytic modeling.Value Health. 2015; 18: 512-529
- Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment.Pharmacoeconomics. 2006; 24: 355-371
- ExpoQual: evaluating measured and modeled human exposure data.Environ Res. 2019; 171: 302-312
- Consolidated health economic evaluation reporting standards (CHEERS) statement.BMJ. 2013; 346: f1049
- GRADE guidelines: 3. Rating the quality of evidence.J Clin Epidemiol. 2011; 64: 401-406
- GRADE guidelines: 6. Rating the quality of evidence--imprecision.J Clin Epidemiol. 2011; 64: 1283-1293
- GRADE guidelines: 8. Rating the quality of evidence--indirectness.J Clin Epidemiol. 2011; 64: 1303-1310
- GRADE guidelines: 7. Rating the quality of evidence--inconsistency.J Clin Epidemiol. 2011; 64: 1294-1302
- GRADE guidelines: 5. Rating the quality of evidence--publication bias.J Clin Epidemiol. 2011; 64: 1277-1282
- GRADE guidelines: 9. Rating up the quality of evidence.J Clin Epidemiol. 2011; 64: 1311-1316
- GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias).J Clin Epidemiol. 2011; 64: 407-415
- Chapter I: Introduction.in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions. 6.1. Cochrane, 2020https://training.cochrane.org/handbook/current/chapter-iDate accessed: October 13, 2020
- System identification: parameter and state estimation.Wiley-Interscience, London1974
- Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.CMAJ. 2003; 169: 677-680
- GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.J Clin Epidemiol. 2016; 76: 89-98
- Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.BMJ. 2008; 336: 1106-1110
- Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients.BMJ. 2015; 350: h870
- Facilitating healthcare decisions by assessing the certainty in the evidence from preclinical animal studies.PLoS One. 2018; 13: e0187271
- GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence.J Clin Epidemiol. 2013; 66: 140-150
- GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness.J Clin Epidemiol. 2018;
- GRADE guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences-inconsistency, imprecision, and other domains.J Clin Epidemiol. 2018;
- WHO guidelines for screening and treatment of precancerous lesions for cervical cancer prevention.World Health Organization, Geneva, Switzerland2013
- Using GRADE to respond to health questions with different levels of urgency.Environ Int. 2016; 92-93: 585-589
- The use of mathematical modeling studies for evidence synthesis and guideline development: a glossary.Res Synth Methods. 2019; 10: 125-133
- Chapter 5: The reference case. Guide to the methods of technology appraisal 2013.NICE, 2013https://www.nice.org.uk/process/pmg9/Date accessed: October 13, 2020
- Food pricing strategies, population diets, and non-communicable disease: a systematic review of simulation studies.PLoS Med. 2012; 9: e1001353
- Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.Value Health. 2014; 17: 174-182
- Appendix G: Methodology checklist: economic evaluations. The guidelines manual: Process and methods.NICE, 2012https://www.nice.org.uk/process/pmg6/Date accessed: October 13, 2020
- Assessing uncertainty in read-across: questions to evaluate toxicity predictions based on knowledge gained from case studies.Comput Toxicol. 2019; 9: 1-11
- Identification and description of the uncertainty, variability, bias and influence in quantitative structure-activity relationships (QSARs) for toxicity prediction.Regul Toxicol Pharmacol. 2019; 106: 90-104
- Identification, review, and use of health state utilities in cost-effectiveness models: an ISPOR good practices for outcomes research task force report.Value Health. 2019; 22: 267-275
- NICE DSU Technical Support Document 13: Identifying and Reviewing Evidence to Inform the Conceptualisation and Population of Cost-Effectiveness Models.National Institute for Health and Care Excellence, London, UK2011
- Identification of evidence for key parameters in decision-analytic models of cost effectiveness: a description of sources and a recommended minimum search requirement.Pharmacoeconomics. 2016; 34: 597-608
- GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes.J Clin Epidemiol. 2013; 66: 151-157
- Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide.Med Decis Making. 2011; 31: 675-692
- Global sensitivity analysis. The primer.John Wiley & Sons, Chichester, England2008
- Chapter 13: Assessing risk of bias due to missing results in a synthesis.in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions. 6.1 Cochrane, 2019https://training.cochrane.org/handbook/current/chapter-13Date accessed: October 13, 2020
- Breast cancer screening and Diagnosis: a synopsis of the European Breast guidelines.Ann Intern Med. 2020; 172: 46-56
- HIV treatment as prevention: systematic comparison of mathematical models of the potential impact of antiretroviral therapy on HIV incidence in South Africa.Plos Med. 2012; 9: e1001245
- Conducting systematic reviews of economic evaluations.Int J Evid Based Healthc. 2015; 13: 170-178
- Collaborative modeling of the benefits and harms associated with different U.S. Breast cancer screening strategies.Ann Intern Med. 2016; 164: 215-225
- Centre for the Mathematical Modelling of Infectious Diseases C-wg. Effects of non-pharmaceutical interventions on COVID-19 cases, deaths, and demand for hospital services in the UK: a modelling study.Lancet Public Health. 2020; 5: e375-e385
- Dermal absorption of chemicals: estimation by IH SkinPerm.J Occup Environ Hyg. 2014; 11: 19-31
- Comparison of four probabilistic models (CARES((R)), Calendex, ConsExpo, and SHEDS) to estimate aggregate residential exposures to pesticides.J Expo Sci Environ Epidemiol. 2012; 22: 522-532
United States Environmental Protection Agency. Human Exposure Modeling - Overview. https://www.epa.gov/fera/human-exposure-modeling-overview. Accessed October 13, 2020.
- Hopscore: An Electronic Outcomes-Based Emergency Triage System.Agency for Healthcare Research and Quality, 2018https://digital.ahrq.gov/sites/default/files/docs/citation/r21hs023641-levin-final-report-2018.pdfDate accessed: October 13, 2020
- The economy-wide impact of pandemic influenza on the UK: a computable general equilibrium modelling experiment.BMJ. 2009; 339: b4571
- The GRADE Working Group clarifies the construct of certainty of evidence.J Clin Epidemiol. 2017; 87: 4-13
Article info
Publication history
Published online: September 24, 2020
Accepted:
September 17,
2020
Identification
Copyright
© 2020 Published by Elsevier Inc.
