If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Centre for Health Economics, Alcuin 'A' Block, University of York, Heslington, York YO10 5DD, UKDepartment of Economics and Related Studies, University of York, York YO10 5DD, UK
To establish whether evidence about the effectiveness of a health care intervention is sufficient to justify the use of the intervention in practice and show how value of information (VOI) analysis can be used to place a value on the need for additional evidence and inform research prioritization decisions.
Study Design and Setting
Meta-analysis provides an estimate of the effect of an intervention with uncertainty. VOI analysis determines the adverse health consequences of not resolving this uncertainty. A case study examining the evidence before the high profile trial of Corticosteroid Randomisation After Significant Head injury (CRASH) shows the consequences on patient outcomes if this trial had not been successfully funded.
Results
The consequences of uncertainty before CRASH were high at 40 deaths and 1,067 years of full health per annum. VOI analysis indicates that CRASH was worthwhile and the UK National Health Service would have had to spend an additional £205 million elsewhere to generate health benefits similar to CRASH.
Conclusions
VOI analysis can be integrated with the results of meta-analysis to help inform whether a particular research proposal is potentially worthwhile and whether it should be prioritized over other research topics that could be commissioned with the same resources.
A simple extension of standard meta-analysis with methods of value of information (VOI) analysis provides an estimate of the health benefits of further research, which can be used to inform research prioritization and commissioning decisions.
What this adds to what was known?
•
Until now, methods of VOI analysis have been applied to situations where probabilistic decision analytic models or estimates of cost effectiveness are available, but we show that the same principles and methods are relevant to a range of different types of health care systems and decision-making contexts, even those where there is no explicit assessment of cost effectiveness.
What is the implication and what should change now?
•
Research proposals that are likely to be worthwhile can be identified in a systematic way. The health benefits expected from different proposals, competing for the same resources, can be compared to each other to establish which topic offers the greatest value. This adds transparency and accountability to research prioritization decisions.
The evidence about the effectiveness of a health care intervention might suggest that it achieves better health outcomes than the available alternative interventions. However, the estimate of treatment effect may still be uncertain, which creates uncertainty in any decision about whether to use the intervention in clinical practice. If the expected health benefits of the intervention are not realized in practice, there may be a detrimental effect to patient health outcomes. In addition, the resources committed by the use of the intervention may be wasted. Similarly, if an intervention is not expected to perform better than the available alternatives, rejecting its use in clinical practice may risk failing to provide access to a valuable intervention if the health benefits are actually greater than expected. These uncertainties can never be entirely eliminated, but they can be reduced by collecting further evidence, which in turn facilitates better decisions for patient outcomes and better use of finite resources.
Value of information (VOI) analysis provides a very useful tool for establishing: (1) whether the evidence currently available is sufficient to support the use of the intervention in practice; (2) whether additional evidence is required to resolve the uncertainties; (3) the type of evidence that is required; and (4) the circumstances under which an intervention should be withheld until additional evidence becomes available [
Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development.
]. There are now many applications of VOI analysis in the context of decision models used to estimate the cost effectiveness of alternative interventions. In this article, we show that the same type of analysis can also be applied to standard results of meta-analysis, without the necessity to undertake a cost-effectiveness analysis. Furthermore, the methods provide a framework to assess the relative importance of alternative research topics and proposals, which is invaluable for research prioritization decisions.
Meta-analysis provides an estimate of the magnitude of treatment effect and the level of uncertainty in this estimate, for example, the confidence interval (CI) around the mean estimate of effect is used to represent the range of values in which the unknown “true” effect lies [
]. When this uncertainty is combined with information about baseline risk and incidence, the absolute effect of the uncertainty on health outcomes can be assessed [
]. VOI analysis determines an estimate of the health benefits that could be gained if the uncertainty about treatment choice was resolved completely. These health benefits can then be compared with the costs of undertaking the research to establish whether it represents an efficient use of resources. Furthermore, the health benefits of different research topics (or proposals for funding) can be compared to establish which topic should be prioritized from those competing for the same resources.
This article shows how the methods of VOI analysis can be integrated with the results of meta-analysis to directly inform the questions posed in research prioritization and commissioning decisions. We take as a starting point that research proposals will include a systematic review of existing evidence and, where appropriate, a meta-analysis because funding additional research without knowledge of existing evidence would seem inappropriate and potentially unethical. We use the case of corticosteroids after traumatic brain injury (TBI) to offer a demonstration of the ease with which the methods can be applied.
2. Corticosteroids after TBI
Despite 19 randomized controlled trials before the CRASH trial (Corticosteroid Randomisation After Significant Head injury) [
CRASH trial collaborators Effect of intravenous corticosteroids on death within 14 days in 10008 adults with clinically significant head injury (MRC CRASH trial): randomised placebo-controlled trial.
CRASH trial collaborators Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury—outcomes at 6 months.
], the effect of corticosteroids on death and disability after TBI remained unclear. The CRASH trial was stopped early after enrolling 10,008 adults with TBI. It reported a higher risk of death or severe disability associated with the use of corticosteroids compared with not using them [
CRASH trial collaborators Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury—outcomes at 6 months.
]. As a consequence of this definitive, and to some extent, unexpected result clinical practice changed dramatically, resulting in many thousands of deaths averted around the world (before CRASH, corticosteroids was used in 64% of patients with TBI in the United States [
]). The global value of the CRASH trial appears, with hindsight, self-evident. However, the prevention of thousands of unnecessary iatrogenic deaths hinges on the fact that the funding application for CRASH was successful. In this article, we conduct a retrospective analysis of the evidence available before CRASH to show how methods of VOI analysis would have been useful for quantifying the value of obtaining further evidence and the expected health consequences of not obtaining the evidence.
2.1 Evidence available before CRASH
The evidence from the trials comparing the use of corticosteroids to placebo or no treatment in acute TBI before CRASH is illustrated in Fig. 1 for the primary end point of mortality. These trials dating from 1972 to 1995 were of varying study quality, length of follow-up, steroids administered, doses, and time to administration [
]. For example, a random-effects meta-analysis suggests that the use of steroids after TBI reduces the risk of death with an expected odds ratio (OR) of 0.93. However, the 95% CI crosses the line of no difference indicating that the change in the risk of death could be as much as 12.5% lower to 9.9% higher (using the average pooled death rate in the control arms of 35.3%).
Fig. 1Fixed and random-effects meta-analysis of the effect of corticosteroids on mortality. OR, odds ratio; CI, confidence interval.
Uncertainty in the effect of corticosteroids on mortality means that there is a chance that any decision about their use in TBI will be incorrect. The impact of this uncertainty on number of deaths per annum is obtained by sampling from the uncertain distributions of relative effect and baseline risk (estimated in the meta-analysis) and multiplying by the annual incidence of TBI. For example, if we sampled a relative effect of 0.90 and a baseline risk of 0.35 from the uncertain distributions for treatment effect and baseline risk of mortality, respectively, the number of deaths per annum for an incidence of 1,000 is 315 (=0.90 × 0.35 × 1000) in the intervention group compared with 350 (=0.35 × 1000) in the control. In this case, the best treatment choice is the intervention as it results in 35 fewer deaths per annum. This process of sampling from the uncertain distributions is repeated many times (e.g., 5,000 times) to derive a distribution of the consequences of uncertainty in number of deaths per annum.
The resulting distribution of the number of deaths per annum based on the evidence before CRASH is illustrated in Fig. 2 for an annual incidence of 8,800 in United Kingdom. If the use of corticosteroids before CRASH had been based on the random-effect analysis which favored their use (OR < 1.0), then there was a 74% chance that corticosteroids were effective and improved survival. However, there was a 26% chance that using corticosteroids in TBI would result in excess deaths. Fig. 2 shows that there was a greater likelihood of small numbers of excess deaths (e.g., 19% chance of greater than 0 and ≤200 deaths per year) and a smaller likelihood of larger numbers of excess deaths (e.g., 7% chance of >200 deaths per year). The average over the distribution in Fig. 2 is the expected (average) number of deaths per annum (40 deaths) due to the uncertainty in the use of corticosteroids after TBI. This value represents the maximum expected health benefits that could be gained if the uncertainty about the effectiveness of corticosteroids was resolved completely, that is, it represents an upper bound on the value of additional evidence to resolve this uncertainty.
Fig. 2Distribution of the consequences of uncertainty in number of deaths per annum based on a random-effects meta-analysis of the effect of corticosteroids on mortality after traumatic brain injury (TBI). There is a 74% chance that corticosteroids are effective and improve survival (i.e., a probability of 0.74 of no deaths). However, because of the uncertainty in the effect of corticosteroids on mortality, there is a 26% chance that corticosteroids will result in excess deaths compared with not using corticosteroids after TBI (control group). The number of excess deaths (x-axis) is shown by the likelihood of them occurring (y-axis). For example, the second bar corresponding to [1,50] with a probability of 0.07 represents a 7% chance that the number of excess deaths from corticosteroid use is between 1 and 50 deaths per annum.
Before CRASH, clinical practice in the United Kingdom did not reflect this balance of evidence that favored the use of corticosteroids, partly due to the substantial uncertainty about its effectiveness. Before CRASH, approximately 12% of patients with TBI received corticosteroids [
]. Therefore, the value of both implementing the uncertain findings of existing research and acquiring additional evidence is greater at 180 deaths per annum.
4. Value of additional evidence about survival and disability
Mortality is only one aspect of outcome in TBI because the impact on disability and subsequent survival is also important. Although the primary outcome reported in 16 of the trials was number of deaths [
], which categorizes people into one of five health states: (1) dead; (2) persistent vegetative; (3) severe disability; (4) moderate disability; and (5) recovery, was also used to assess neurological outcomes in seven of the trials [
]. The meta-analysis can therefore be extended to include the effects of corticosteroids on both survival and disability. Table 1 shows the proportion of individuals expected to be in each of the GOS outcomes based on a random-effects analysis of all outcomes. The risk of being left in a vegetative or severely disabled state was higher with the use of corticosteroids compared with control, whereas the risk of being left moderately disabled or making good recovery was lower with corticosteroids.
Table 1Proportion of individuals in each of the Glasgow Outcome Scale (GOS) health states based on a random-effects meta-analysis of the effect of corticosteroids on survival and disability after traumatic brain injury
Seven of the 16 trials reported outcomes in terms of the GOS [16,18–23], whereas a further two trials reported the combined effect on death, vegetative, and severe disability [24,25].
The conventional scale for health-related quality of life weights is between 0 (death) and 1 (perfect health). One year in a health state with a weight of 0.5 is equivalent to half a year in full health. One study was identified which provided the health-related quality of life weights by GOS outcomes [34].
Quality-adjusted life expectancy quantifies an individual's remaining life expectancy in terms of years lived in full health by weighing survival in worse health states lower than survival in full health. One study was identified that estimated life expectancy in traumatic brain injury by age and severity of disability and taking account of any change in health status over a person's lifetime [35].
Corticosteroids
No corticosteroids
Dead
33.5 (22.8, 45.2)
35.3 (24.8, 46.9)
0.00
0.00
Vegetative
4.8 (2.8, 7.5)
3.8 (2.4, 5.9)
0.08 (0.16)
0.56
Severe disability
13.5 (8.3, 20.1)
10.7 (7.1, 15.8)
0.26 (0.25)
3.24
Moderate disability
11.6 (8.6, 14.8)
12.1 (9.2, 15.1)
0.63 (0.27)
10.51
Good recovery
36.5 (28.1, 44.8)
38.0 (30.1, 45.6)
0.85 (0.19)
15.39
Abbreviations: CI, confidence interval; SE, standard error.
a Seven of the 16 trials reported outcomes in terms of the GOS
b The conventional scale for health-related quality of life weights is between 0 (death) and 1 (perfect health). One year in a health state with a weight of 0.5 is equivalent to half a year in full health. One study was identified which provided the health-related quality of life weights by GOS outcomes
c Quality-adjusted life expectancy quantifies an individual's remaining life expectancy in terms of years lived in full health by weighing survival in worse health states lower than survival in full health. One study was identified that estimated life expectancy in traumatic brain injury by age and severity of disability and taking account of any change in health status over a person's lifetime
When these important aspects of outcome are considered, the judgment about the effectiveness of corticosteroids and the uncertainty is changed. For example, when the evidence for the worse health outcomes of death, vegetative, and severely disabled were combined, the OR was 1.10 (95% CI: 0.81, 1.53) against the use of corticosteroids. By exploiting other external evidence, the outcome of survivors can be quantified in terms of their subsequent quality of life. Shavelle et al. [
] estimated the expected life expectancy of an individual after TBI by age and severity of disability (taking account of any change in health status over the individual's lifetime). To quantify an individual's remaining life expectancy in terms of years lived in full health, health-related quality of life weights are used to weight survival in worse health states lower than survival in full health. Table 1 also shows the expected number of years lived in full health and corresponding health-related quality of life weights for the GOS outcomes [
The distribution of the overall health consequences of uncertainty associated with the effectiveness of corticosteroids is illustrated in Fig. 3 for number of years lived in full health per annum. On the balance of evidence about total health outcomes (not just mortality), corticosteroids were not expected to be effective (OR > 1.0). Therefore, Fig. 3 describes the consequences of uncertainty surrounding the decision to not use corticosteroids in practice. It shows that there was a 63% chance that this was the correct decision, that is, it did not result in a loss of years lived in full health. However, there was a 37% chance that corticosteroids could have improved health outcomes (including a 23% chance of between 0 and 3,000 years of full health gained per year and 14% chance of >3,000 years of full health gained per year). The average over the distribution in Fig. 3 of 1,067 years of full health each year represents the health gains of resolving the uncertainty through gathering additional evidence (e.g., a large randomized trial to address the uncertainty in outcomes after TBI).
Fig. 3Distribution of the consequences of uncertainty in number of years lived in full health per annum based on a random-effects meta-analysis of the effect of corticosteroids on survival and disability after traumatic brain injury (TBI). There is a 63% chance that corticosteroids are not effective at improving survival and disability. However, because of the uncertainty in the effect of corticosteroids on survival and disability, there is a 37% chance that a decision to not use corticosteroids is wrong. The number of lost years in full health as a consequence of this uncertainty is shown on the x-axis, whereas the y-axis shows the likelihood of these consequences occurring. For example, the second bar corresponding to [1,1000] with a probability of 0.09 represents a 9% chance that the number of years of full health lost from not using corticosteroids after TBI is between 1 and 1,000 per annum.
At the time, 12% of patients with TBI received corticosteroids in the United Kingdom. Therefore, the value of both implementing the uncertain findings of existing research (not to use corticosteroids for TBI) and acquiring additional evidence that would resolve this uncertainty was greater at 1,264 years of full health gained each year.
5. Assessing the value and priority of proposed research
Two questions are posed when considering whether the decision to prioritize and commission CRASH was appropriate:
1.
Are the benefits of additional evidence of 1,264 years of full health per year sufficient to regard CRASH as potentially worthwhile?
2.
Should CRASH be prioritized over other research topics that could be commissioned with the same resources?
These assessments require a judgment of the period of time over which the additional evidence is likely to be relevant and the expected time it takes for the research to be commissioned, conducted, and report. Information generated by research will not be valuable indefinitely because of other changes occurring over time [
Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development.
]. For example, new and more effective interventions may become available, making current comparators obsolete and rendering information about their effectiveness irrelevant to future clinical practice. The actual time horizon for evidence generated by a particular research proposal is unknown because it is a proxy for a complex and uncertain process of future changes [
]. However, some judgment is unavoidable when making decisions about research priorities.
The CRASH trial was proposed to the UK Medical Research Council (MRC) in the year 2000. A time horizon of 15 years may have been a reasonable but conservative judgment at the time, given that there were no other trials underway and previously few major innovations which had transformed the treatment or understanding of TBI. The implications for an assessment of the overall expected benefits of CRASH are illustrated in Fig. 4.
Fig. 4The value of additional evidence based on a random-effects meta-analysis of the effect of corticosteroids on survival and disability after traumatic brain injury (TBI) before the definitive trial of CRASH was commissioned. The CRASH trial was proposed in the year 2000 but was not expected to report before the year 2004. The value of resolving the uncertainty in the effect of corticosteroids on survival and disability was 10,266 years of full health in the year 2000, with health benefits discounted at a rate of 3.5% per annum (or 13,904 years lived in full health without discounting). CRASH, Corticosteroid Randomisation After Significant Head injury.
CRASH was not expected to report before the year 2004. Therefore, the overall (undiscounted) expected health benefits were an additional 13,904 years lived in full health (1,264 years of full health per annum × 11 years from 2004 to 2015). In the UK context, both health benefits and National Health Service (NHS) costs are discounted at a rate of 3.5%, so it is the discounted value of 10,266 years of full health that is most relevant. The question remains, however, whether these expected benefits were sufficient to justify the expected costs of the CRASH trial (£2.2 million) and whether it represented a particular priority compared with the other research that could have been commissioned by the MRC using the same resources?
One way to address this question in the United Kingdom is to ask whether the NHS could have generated similar expected heath gains more effectively elsewhere or whether the costs of the CRASH trial would have generated more health benefits if these resources had been made available to the NHS. Recent work in the United Kingdom [
] has estimated the relationship between changes in NHS expenditure and health outcomes. This work suggests that the NHS spends approximately £75,000 to avoid one premature death, £25,000 to gain one life year, and £20,000 to gain one quality-adjusted life year [
]. Using these estimates, the costs of CRASH could have been used to avoid 29 deaths (=£2.2 m/£75,000) and generate 110 quality-adjusted life years (=£2.2 m/£20,000) elsewhere in the NHS—substantially less than the expected health benefits of the CRASH trial. Alternatively, the NHS would have to spend an additional £205 million (=10,266 years of full health × £20,000 for 1 year of full health) between 2004 and 2015 to generate expected health benefits similar to those offered by CRASH. This strongly suggests that the CRASH trial was indeed worthwhile at the time it was commissioned.
However, because the MRC itself has limited resources and cannot draw directly on the NHS budget, it is possible that other research proposed in the year 2000 may have been even more valuable than CRASH. Without a similar reanalysis of rejected research proposals at the time, it is not possible to confirm that CRASH offered the greatest value. If similar analysis was conducted for all topics competing for limited research resources, it does become possible to identify a shortlist of those which are likely to be worthwhile.
It is useful to reconsider the analysis set out above once the results of CRASH became available by updating the meta-analysis and the estimates of the expected potential benefits of further research. When the results of the CRASH trial are included in the meta-analysis, the chance that corticosteroids improves mortality is reduced to 0 (<0.0001), whereas the chance that corticosteroids improves survival and quality of life is almost 0 (probability of 0.005). Therefore, when the analysis of the potential value of additional evidence is updated, there are no expected benefits of acquiring additional evidence in TBI. Therefore, CRASH was a definitive trial, appropriately prioritized and commissioned at the time.
6. Discussion
The above analysis has shown how methods of VOI analysis can be integrated with the results of meta-analysis to inform the assessments that are required when making decisions about research priorities. The application of VOI analysis to results from a standard meta-analysis is not technically challenging nor does it pose any particular computational problems. However, some of the contexts to be examined may require more sophisticated forms of meta-analysis (e.g., Bayesian meta-analysis to link multiple end points) and careful consideration of the relevance and quality of evidence (e.g., through the use of the Grading of Recommendation, Assessment, Development, and Evaluation [GRADE] approach [
]). Nevertheless, it could be argued that these methods would generally be required anyway to estimate the effectiveness of the interventions even before the value of additional evidence was considered.
In some circumstances, the end points included in the meta-analysis of studies may not capture all valuable aspects of health outcome; for example, mortality as a primary outcome after TBI was shown above to not necessarily be the only relevant outcome. In these circumstances, it may be possible to use external evidence to link the end points to other aspects of health outcome, for example, quality-adjusted life years. In other circumstances, it may be necessary to specify a minimum clinical difference in outcomes (effect size) required to change clinical practice as a means of incorporating concerns that there are other important aspects of outcome (e.g., adverse events or quality of life impacts that have not been accounted for in the meta-analysis). Requiring that further research must demonstrate larger differences in effect will tend to reduce the expected benefits of research because larger differences are less likely to be observed than smaller ones. Importantly, whatever the policy context, the principles and established methods of VOI analysis are relevant to a wide range of different types of health care systems and decision-making contexts and should not be regarded as being restricted to situations where probabilistic decision analytic models or estimates of cost effectiveness are available.
Of course, no quantitative analysis can capture all aspects of scientific and social value judgments about research priorities. Therefore, the most relevant question is whether these methods offer a practical and useful starting point for deliberation and add to the transparency and accountability of research prioritization decisions.
References
Basu A.
Meltzer D.
Value of information on preference heterogeneity and individualized care.
Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development.
Effect of intravenous corticosteroids on death within 14 days in 10008 adults with clinically significant head injury (MRC CRASH trial): randomised placebo-controlled trial.
Funding: This research was supported by The Patient-Centered Outcomes Research Institute in the United States. H.K. was also supported by the Netherlands Organisation for Scientific Research (grant # 916.11.126).
00437-0&id=gr1.jpg){kind=link}
00437-0&id=gr2.jpg){kind=link}
00437-0&id=gr3.jpg){kind=link}
00437-0&id=gr4.jpg){kind=link}