There has never been a time when evidence was more important or under such intense scrutiny. A time when every key study is eagerly awaited and devoured on arrival as though the health and hopes of the global community rest on its shoulders. And yet, as we know, evidence is frequently flawed, equivocal in its findings and open to interpretation, even though it forms a critical element of most guidelines and decision support tools. Over recent years the concept of a continuous evidence eco-system that comprises different but mutually dependent processes, a broad range of actors and stakeholders, and the increased exploitation of systems and technology has been developed [
[1]
]. In this month's issue of the JCE we present articles that address challenges that have been identified at three key points of the evidence eco-system, namely primary research, evidence synthesis and translation into guidelines [2
, 3
, 4
, 5
, 6
].Randomised controlled trials continue to represent the best primary research tools we have to evaluate the effects of healthcare interventions. However, as Nguyen and Xie show, even when randomisation processes are adequate, a substantial proportion of studies (about a third in their study) include at least one baseline variable that might, by chance, be substantially unbalanced across treatment groups [3]. The authors report that clinical trial reports frequently fail to describe or assess post hoc measures of comparability. Even where tests for comparability are undertaken, they are frequently flawed. They describe a potential solution that uses a measurement of the standard mean difference between treatment groups to assess baseline difference. This technique enables characteristics initially expressed in different units to be viewed on a similar scale. This is not simply a dry methodological issue: in the context of the COVID-19 pandemic, the issue of baseline difference has been a key source of concern for those tasked with interpreting the data. This can span across the usual parameters of age, sex, setting, stage of disease and co-morbidity, but also the vexed issue of multiple co-interventions, often of unknown effects and frequently given on purely compassionate grounds.
Even a perfectly conducted study where the risk of bias is scrupulously minimised can be undone by reporting that does not fairly or accurately reflect its findings. Spin, or ‘misrepresentation, overinterpretation or inappropriate extrapolation of results’, is less well studied than bias, but its frequency and importance in the reports of both primary research and evidence synthesis should not be underestimated. Ghannad and colleagues have undertaken what they believe may be the first RCT of an intervention aimed at reducing spin [4]. They identify four key elements of spin: selective reporting, information that is not supported by the evidence, interpretation that is inconsistent with the study design or results, and recommendations for practice that are not justified by the study's findings. They implemented a simple editorial intervention and compared this with standard practice. Perhaps unsurprisingly, none of the results demonstrated a statistically significant benefit, although confidence intervals were fairly wide, reflecting uncertainty rather than evidence of no effect. Thus, for the time being, the task of policing spin will continue to sit with editorial teams. We hope that further research will follow, perhaps even extending as far as the press releases from studies, which are currently assuming an inappropriate and unwarranted importance in our discipline.
At the far end of the evidence eco-system process is the work undertaken by organisations and groups to factor the best current evidence alongside other critical issues to develop clinical guidelines. Contributing as part of a guideline development group is a heavy responsibility, so it seems surprising that until the paper by Piggott and colleagues included in this issue, there was little guidance for members of such groups [5]. The paper describes a mixed methods and iterative study undertaken to develop a guideline participation tool. The work was undertaken by an international group of researchers who have extensive experience as members of the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) working group. The tool should be helpful for all participants, but particularly for those with limited experience for whom the process must seem particularly challenging. Importantly, the work will be incorporated into guideline training supported by the Guidelines International Network. The result should be better prepared guideline group members, greater inclusiveness and engagement in group meetings and improved outputs.
Piggott and his colleagues present a second paper that addresses an emerging issue in relation to the application of multiple comparison analyses to the GRADE Evidence to Decision (EtD) framework, for use by guidelines groups [6]. Network meta-analysis has become a preferred approach to ranking multiple treatment options, utilising as it does, indirect and direct evidence. However, as the researchers describe, this only addresses the dimensions of benefits and harms, whereas the EtD framework typically also incorporates certainty, values, resources, equity, acceptability and feasibility. The work presented here represents the output of a working group working across three distinct scenarios, and includes the incorporation of a module based on the developments into the GRADE Pro application. The result is a structured approach that supports full transparency of the evidence and decision making across multiple comparisons. The process begins with a full EtD evaluation for each of the relevant pairwise comparisons, and an assessment of the transitivity of the interventions being studied. This facilitates ranking of the interventions across the EtD criteria, which may be used to inform recommendations.
The studies presented here, and others included in this issue demonstrate how researchers and other relevant actors are addressing the critical challenges that can undermine the evidence eco-system. By the application of the empirical solutions presented, health policy and clinical care for our populations move forward.
References
- Future of evidence ecosystem series: evidence ecosystems and learning health systems: why bother?.J Clin Epidemiol. 2020; 123: 166-170
- Future of evidence ecosystem series: 1. Introduction — evidence synthesis ecosystem needs dramatic change.J Clin Epidemiol. 2020; 123: 135-142
- Incomparability of treatment groups is often blindly ignored in randomised controlled trials – a post hoc analysis of baseline characteristics tables.J Clin Epidemiol. 2020; 130: 161-168
- A randomized trial of an editorial intervention to reduce spin in the abstract’s conclusion of manuscripts showed no significant effect.J Clin Epidemiol. 2020; 130: 69-77
- Supporting effective participation in health guidelines development groups: the guideline participation tool.J Clin Epidemiol. 2020; 130: 42-48
- Using GRADE evidence to decision frameworks to choose from multiple interventions.J Clin Epidemiol. 2020; 130: 117-124
Article info
Identification
Copyright
© 2020 Published by Elsevier Inc.
