Abstract
Objectives
To assess how much protocols reduce methodological flexibility and variation of results
in meta-analyses comparing operative with nonoperative treatments for proximal humerus
fractures.
Study Design and Setting
A meta-epidemiological study. We searched four bibliographic databases for eligible
meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of
their protocol. We identified the included studies and extracted intervention effect
data for functional outcome, quality of life, and adverse events. Using the same intervention
effect data for each outcome domain, we conducted multiple meta-analyses using differing
combinations of methodological choices (analytical scenarios) without restricting
the available methodological choices (as if no protocol was used). For each protocol,
we repeated this process but restricted the available choices to those listed in the
protocol. We then compared the distributions of effect estimates with and without
protocols. Methodological flexibility was estimated as the difference in number of
possible meta-analyses and the variation of results as the difference between the
most conflicting effect estimates.
Results
We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols.
The protocols markedly reduced methodological flexibility (range for functional outcomes
94%–99%; quality of life 58%–76%; adverse events 87%–91%). The corresponding reduction
in the variation of the results varied (range for functional outcomes; 33%–78%, quality
of life; 10%–24%; adverse events 10%–13%).
Conclusion
Although the protocols consistently reduced methodological flexibility, the reduction
in the variation (bias) of results varied. Consequently, review authors may obtain
conflicting results based on the same data. We advise caution when interpreting the
conclusions of meta-analyses with absent or imprecise protocols.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- A manifesto for reproducible science.Nat Hum Behav. 2017; 1: 1-9
- Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy.J Clin Epidemiol. 2017; 91: 95-110
- Why most discovered true associations are inflated.Epidemiology. 2008; 19: 640-648
- Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations.J Clin Epidemiol. 2015; 68: 1046-1058
- Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis.BMC Med. 2019; 17: 1-3
- The methodological quality was low and conclusions discordant for meta-analyses comparing proximal humerus fracture treatments: a meta-epidemiological study.J Clin Epidemiol. 2022; 142: 100-109
- A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects.J Clin Epidemiol. 2022; 152: 140-150
- Vibration of effect in more than 16 000 pooled analyses of individual participant data from 12 randomised controlled trials comparing canagliflozin and placebo for type 2 diabetes mellitus: multiverse analysis.BMJ Med. 2022; 1
- Why most published research findings are false.PLoS Med. 2005; 2: e124
- False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant.Psychol Sci. 2011; 22: 1359-1366
- Surgical and non-surgical treatment for 3-part and 4-Part Fractures of the proximal humerus: a systematic review of overlapping meta-analyses.Orthop Surg. 2019; 11: 356-365
- Cochrane handbook for systematic reviews of interventions.2nd ed. John Wiley & Sons, Chichester (UK)2019
- Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.PLoS Med. 2016; 13: e1002028
- Chapter 8: assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0.(Available at)https://training.cochrane.org/handbook/archive/v5.1/Date accessed: March 14, 2023
- RoB 2: a revised tool for assessing risk of bias in randomised trials.BMJ. 2019; 366: l4898
- Methodological index for non-randomized studies (minors): development and validation of a new instrument.ANZ J Surg. 2003; 73: 712-716
- The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses.(Available at)https://www3.med.unipmn.it/dispense_ebm/2009-2010/Corso%20Perfezionamento%20EBM_Faggiano/NOS_oxford.pdfDate accessed: March 14, 2023
- Operative versus nonoperative treatment of proximal humeral fractures: a systematic review, meta-analysis, and comparison of observational studies and randomized controlled trials.J Shoulder Elbow Surg. 2018; 27: 1526-1534
- Conservative versus operative management of complex proximal humeral fractures: a meta-analysis.Shoulder Elbow. 2010; 2: 166-174
- Effectiveness and safety of interventions for treating adults with displaced proximal humeral fracture: a network meta-analysis and systematic review.PLoS One. 2016; 11: e0166801
- Primary shoulder arthroplasty versus conservative treatment for comminuted proximal humeral fractures: a systematic literature review.Open Orthop J. 2010; 4: 87-92
- Interventions for Treating 3- or 4-part proximal humeral fractures in elderly patient: a network meta-analysis of randomized controlled trials.Int J Surg. 2017; 48: 240-246
- Surgical versus conservative treatment for displaced proximal humeral fractures in elderly patients: a meta-analysis.Int J Clin Exp Med. 2014; 7: 4607-4615
- Meta-analysis of joint preservation versus arthroplasty for the treatment of displaced 3- and 4-part fractures of the proximal humerus.Injury. 2013; 44: 1532-1539
- Interventions for treating proximal humeral fractures in adults.Cochrane Database Syst Rev. 2015; 2015: CD000434
- Operative versus nonoperative treatment for complex proximal humeral fractures: a meta-analysis of randomized controlled trials.Orthopedics. 2014; 37: e543-e551
- Internal fixation versus nonoperative treatment for displaced 3-part or 4-Part Proximal humeral fractures in elderly patients: a meta-analysis of randomized controlled trials.PLoS One. 2013; 8: e75464
- Operative versus non-operative treatment for three- or four-part proximal humeral fractures in elderly patients: a meta-analysis of randomized controlled trials.Zhejiang Da Xue Xue Bao Yi Xue Ban. 2016; 45: 641-647
- Comparison of surgical versus non-surgical treatment of displaced 3- and 4-Part Fractures of the proximal humerus: a meta-analysis.J Invest Surg. 2015; 28: 215-224
- Operative versus nonoperative treatment in complex proximal humeral fractures.Orthopedics. 2014; 37: e410-e419
- Treatment of humerus fractures in the elderly: a systematic review covering effectiveness, safety, economic aspects and evolution of practice.PLoS One. 2018; 13: e0207815
- Comparison of surgical and non-surgical treatments for 3- and 4-part proximal humerus fractures: a network meta-analysis.Shoulder Elbow. 2020; 12: 99-108
- Operative vs non-operative management of displaced proximal humeral fractures in the elderly: a systematic review and meta-analysis of randomized controlled trials.World J Orthop. 2015; 6: 838-846
- Trials based on specific fracture configuration and surgical procedures likely to be more relevant for decision making in the management of fractures of the proximal humerus.Bone Joint Res. 2016; 5: 470-480
- Surgical or non-surgical treatment of traumatic skeletal fractures in adults: systematic review and meta-analysis of benefits and harms.Syst Rev. 2020; 9: 179
- Operative vs. Nonoperative treatment for comminuted proximal humeral fractures in elderly patients: a current meta-analysis.Acta Orthop Traumatol Turc. 2015; 49: 345-353
- Treatment of complex proximal humeral fracture: plate and tension band fixation versus conservative therapy.Int J Clin Exp Med. 2015; 8: 7143-7151
- Operative versus non-operative treatment in complex proximal humeral fractures: a meta-analysis of randomized controlled trials.Springerplus. 2015; 4: 728
- Avascular necrosis in proximal humeral fractures in patients treated with operative fixation: a meta-analysis.J Orthop Surg Res. 2014; 9: 31
- Arthroplasty versus joint preservation for displaced 3- and 4-part proximal humeral fractures: a meta-analysis.Chin J Tissue Eng Res. 2014; 18: 4241-4247
- Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.BMJ. 2011; 343: d4829
Article info
Publication history
Published online: February 21, 2023
Accepted:
February 1,
2023
Footnotes
Funding: N.S. was supported by the Department of Orthopedic Surgery at Zealand University Hospital and by Region Zealand and the Region of Southern Denmark's research fund. The funders had no role in the design of the study, collection, analysis, and interpretation of data or in writing the manuscript.
Declaration of interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. S.B. was a co-author of an included index meta-analysis.
Author Contributions: Nicolai Sandau: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing–original draft, Writing–review and editing, Visualization, Supervision, Project administration, and Funding acquisition. Thomas Vedste Aagaard: Conceptualization, Methodology, Investigation, and Writing–review and editing. Asbjørn Hróbjartsson: Conceptualization, Methodology, Writing–review and editing, and Supervision. Ian A Harris: Conceptualization, Methodology, Writing–review and editing, and Supervision. Stig Brorson: Conceptualization, Methodology, Writing–original draft, Writing–review and editing, Supervision, and Funding acquisition.
Identification
Copyright
© 2023 Elsevier Inc. All rights reserved.
