| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Evidence-based Practice |
1 From the Program for the Assessment of Radiological Technology (B.G.K., J.J.N., E.H.O., M.G.M.H.), Department of Epidemiology and Biostatistics (B.G.K., J.J.N., E.H.O., M.G.M.H.), and Department of Radiology (J.J.N., E.H.O., A.Z.G., M.G.M.H.), Erasmus Medical Center, University Medical Center Rotterdam, Dr Molewaterplein 40, Room Ee 21-40a, 3015 GD Rotterdam, the Netherlands; Department of Medical Statistics and Bioinformatics, Leiden University Medical Center, Leiden, the Netherlands (T.S.); and Department of Health Policy and Management, Harvard School of Public Health, Boston, Mass (M.G.M.H.). From the 2004 RSNA Annual Meeting. Received January 14, 2007; revision requested March 15; revision received April 24; accepted May 8; final version accepted July 23. Supported by a program grant (904-66-091) from the Netherlands Organization for Scientific Research. Address correspondence to M.G.M.H. (e-mail: m.hunink{at}erasmusmc.nl).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
Materials and Methods: A total of 189 patients (123 male, 66 female; mean age, 33.4 years) were randomly assigned to undergo radiography alone (n = 93) or radiography and MR imaging (n = 96). Institutional review board approval and informed consent (parental consent for minors) were obtained. During 6 months of follow-up, data on quality of life and 39 cost parameters were collected. Value-of-information analysis was used to estimate the expected benefit of future research to eliminate the decision uncertainty that remained after trial completion. In addition, the parameters that were responsible for most of the decision uncertainty were identified, the expected benefits of various study designs were evaluated, and the optimal sample size was estimated.
Results: Only three parameters were responsible for most of the decision uncertainty: number of quality-adjusted life-years, cost of an overnight hospital stay, and friction costs. A study in which data on these three parameters are gathered would have an optimal sample size of 3500 patients per arm and would be expected to result in a societal benefit of 
5.6 million or 70 quality-adjusted life-years.
Conclusion: The optimal study design for use of MR imaging to evaluate acute knee trauma involves a trial in which there are 3500 patients per trial arm, and data on the number of quality-adjusted life-years, cost of an overnight hospital stay, and friction costs are collected.
© RSNA, 2008
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
If study results show no significant differences between the primary outcomes, the researchers invariably conclude that more clinical research is needed to reduce decision uncertainty (2). Uncertainty could result in the adoption of suboptimal medical interventions, which could harm patients or result in inefficient allocation of limited health care funds. More research—for example, another clinical trial—is expected to decrease this uncertainty and benefit patients, save money, or both. However, research is costly, and money spent on one research project cannot be spent on another. Furthermore, while additional research is being performed, a potentially cost-effective intervention is being withheld from patients. These problems raise the question of whether more research regarding an uncertain decision is a good value for the money. More clinical research is justified only if the expected benefit of this research exceeds the expected cost. Value-of-information analysis is a method that expands on cost-effectiveness analysis and can be used to determine if more research is justified regarding a medical decision. This method is used to estimate the expected benefit of a proposed study given the currently available evidence. In addition, value-of-information analysis can be used to identify the optimal study design and sample size. Use of value-of-information analysis has been embraced and recommended by the National Institute for Clinical Excellence in the United Kingdom as a framework for setting research priorities in health care (3).
The purpose of this study was to help guide future outcomes research regarding the use of MR imaging in patients with acute knee trauma in an emergency department setting, with use of prospective data from a randomized clinical trial and value-of-information analysis.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
During 6 months of follow-up, quality of life was measured four times with a valuative device (EuroQol). All relevant societal costs were recorded during the follow-up period. These costs included medical and nonmedical costs. Medical costs consisted of costs of diagnostic procedures and treatment both inside and outside the hospital and were estimated with 36 resource-use parameters for each strategy. Nonmedical costs were estimated with three parameters: patient travel costs, patient time cost, and friction costs. The latter was an estimate of societal production losses. In total, 40 parameters (39 cost parameters and quality of life) were sampled for each strategy. Mean values and 95% confidence intervals were calculated for costs and effects of both strategies. (See the original article [1] for more details on study design and analysis.)
Cost-effectiveness Analysis
To perform cost-effectiveness analysis, we (B.G.K., J.J.N.) converted Euroqol values into utility values (4). For each patient, an author (B.G.K.) calculated the overall number of quality-adjusted life-years during the study period as the effect parameter (5).
A choice of one of the two strategies that is based on both cost and effect can be made only if a trade-off between cost and effect is made by placing a monetary value on health. We used a societal willingness-to-pay threshold of 80 000 per quality-adjusted life-year, as recently recommended by a Dutch governmental institute (6). Subsequently, we (B.G.K., T.S.) combined cost and effect into one outcome, which we termed net (monetary) benefit (7). Net benefit was calculated by multiplying effect by willingness to pay and subtracting cost. The strategy with the maximum net benefit is the strategy that is preferred.
Value-of-Information Analysis
We (B.G.K., T.S.) applied value-of-information analysis, as described in the literature (8–12). First, we estimated the total expected value of perfect information (EVPI) per patient. This is the value of collecting data about the effect parameter and all cost parameters in an infinitely large study. In other words, it is the value of removing all uncertainty related to the decision problem.
Subsequently, we estimated the expected value for the entire patient population that can potentially benefit from more research (population EVPI). To calculate the population EVPI, we (B.G.K., M.G.M.H.) estimated the effective lifetime of the technology to be 10 years. Benefits to future patients were discounted at a rate of 3% per year (5). For the Netherlands perspective, we estimated the annual population that could potentially benefit from the results of a future study to be 20 000 patients. We performed additional analysis for the European Union perspective. By extrapolating the annual population of 20 000 patients to the European Union, we determined that an annual population of 561 000 patients could benefit from more research. If the population EVPI is substantial, it is of interest to estimate the EVPI for individual parameters or sets of parameters. We termed this the partial EVPI. Partial EVPI is used to identify the parameters that have the highest informational value regarding decision uncertainty.
If the total EVPI is substantial, we are interested to learn the expected benefit of reducing uncertainty by obtaining information from a future study with a finite sample size. This is referred to as the total expected value of sample information (EVSI). Moreover, we can assess the expected benefit of future studies with a finite sample size that is used to collect information on a limited set of parameters. This is referred to as the partial EVSI. An author (B.G.K.) estimated the partial EVSI for several sets of parameters to assess various study designs. Comparing the EVSI with the cost of performing research enables us to determine whether an additional study is justified given the cost. Subtracting the cost of research from the EVSI results in the expected net benefit of sampling (ENBS). The optimal sample size is determined by calculating the sample size that maximizes the ENBS.
For a future multicenter trial with a 3-year duration (assuming the study requires two full-time equivalent junior researchers and a senior researcher with 0.4 full-time-equivalent responsibility), we (B.G.K., M.G.M.H.) assumed a fixed cost of 500 000 and a variable cost of 500 per patient if all parameters in the initial trial were to be measured. If data on only three parameters (friction cost, overnight hospital stay, and quality-adjusted life-years) were to be collected, we assumed a fixed cost of 250 000 and a variable cost of 250 per patient. These cost estimates were based on our current expenses for similar studies.
Technical Details and Analysis
To allow for value-of-information analysis, we (B.G.K., T.S.) represented the joint uncertainty about the mean values of all parameters by using a multivariable normal distribution, with variances equal to the estimated squared standard errors of the mean and correlations between the different parameters calculated from the dataset. The central limit theorem justified the normality assumption.
An author (B.G.K.) performed 10 million simulations for each analysis, resulting in standard errors in our estimates of about 1%. Nested simulations were not required to estimate partial EVPI and EVSI because the relationship between the net benefit and each parameter was linear and because the multivariable normal distribution allowed us to calculate conditional mean values (8,13). To estimate EVSI, an author (B.G.K.) derived posterior normal distributions for the sampled parameters by using Bayesian updating of the prior normal distributions (14). We (B.G.K., T.S.) assumed that the standard deviations and correlations between parameters in future research would be the same as those in the initial trial. All analyses were performed with R software (version 1.7.1; R Foundation for Statistical Computing, Vienna, Austria) that can be accessed at http://www.r-project.org.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
|
2.1 per patient. The resulting population EVPI was 365 000 for the Netherlands and 10.2 million for the European Union. These values have an equivalent benefit of 5 quality-adjusted life-years for the Netherlands and 128 quality-adjusted life-years for the European Union. An effective lifetime for the technology of 5 years instead of 10 years would reduce these benefits by approximately half.
Partial EVPI: Important Parameters
In the initial study, only two of the 40 collected data parameters had a nonzero partial EVPI. The partial EVPI of the quality-adjusted life-year was 1.0 per patient, and the partial EVPI of the friction cost was 0.01 per patient. These two parameters had a synergistic effect, and together they had a partial EVPI of 1.9 per patient. This synergistic effect was augmented by considering the cost of an overnight hospital stay. A future study in which data on the number of quality-adjusted life-years, the cost of an overnight hospital stay, and the friction costs per patient would be gathered would have a partial EVPI of 2.0 per patient, which would be nearly equal to the total EVPI.
Total EVSI and ENBS: Optimal Sample Size
We first considered the optimal sample size for a future study to be identical to that in our previous study, enabling us to collect data on all parameters. The population EVSI for a study from the perspective of the Netherlands did not exceed the study costs for any sample size. This meant that more research was not justified. The population EVSI for a study from the perspective of the European Union increased as the sample size increased until a plateau was reached; this plateau was equivalent to the population EVPI (Fig 1). The study costs increased linearly as the sample size increased. The maximum ENBS of 3.8 million was reached at a sample size of 2500 patients per trial arm. One should note, however, that there was a decreasing marginal gain in the ENBS: A study with 1500 patients per trial arm was expected to reach a net benefit of 3.4 million.
|
5.6 million or 70 quality-adjusted life-years. Again, because of the decreasing marginal gain, we found an ENBS of 5.1 million for a study with 2000 patients.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
10.2 million for a study from the perspective of the European Union regarding the decision of whether to add MR imaging to the current initial work-up of patients with acute knee trauma. This indicates that if we would eliminate all uncertainty regarding this decision, we could expect a societal financial benefit of 10.2 million, which is equivalent to a societal health benefit of 128 quality-adjusted life-years. Only three parameters were responsible for the decision uncertainty: the number of quality-adjusted life-years, cost of an overnight hospital stay, and friction costs per patient. Collecting data on the other 37 cost parameters has almost no additional benefit. A study in which data on these three parameters were gathered would have an optimal sample size of 3500 patients in each trial arm, and it would be expected to result in a societal benefit of 5.6 million or 70 quality-adjusted life-years. From the perspective of the Netherlands, however, more research was not justified.
It is important to realize that the calculated societal benefit of 5.6 million or 70 quality-adjusted life-years is an expected net benefit: It is a probability-weighted average over all possible outcomes of a future study. We learn the actual benefit of a study only after we have initiated the study, collected the data, and analyzed the actual results. Often, there is no actual benefit. The findings of the future study are more likely than not to confirm that the strategy that we believe to be optimal is indeed optimal. If a future study results in a change in the optimal strategy, the benefit may be a reduction in cost, an increase in quality-adjusted life-years, or a combination of these benefits.
The expected societal benefit of 5.6 million should be compared with the expected societal benefit of other unrelated proposed clinical research projects to set research priorities. The decision uncertainty regarding imaging for patients with acute knee trauma turns out to be relatively small in comparison with other clinical problems that have been addressed in value of information analyses (15). More research regarding MR imaging in patients with acute knee trauma is justified, but other clinical studies are expected to result in up to a 100-fold higher benefit. The prioritization of research studies will ultimately depend on the portfolio of potential studies submitted to a funding agency, their corresponding expected value of information, and the available research budget.
Our results were sensitive to the uncertain magnitude of the population expected to benefit from reducing decision uncertainty. This is, by definition, true for all value-of-information analyses; however, it is not a drawback but rather inherent to the assessment of the expected benefit of future research. Both the annual population that can potentially benefit from the research and the effective lifetime of the technology are influential and uncertain. The annual population that can benefit from research depends on the perspective of the policy maker: For example, is it the perspective of the hospital, the state, the country, or something even larger? When research proposals are compared, they need to be judged and compared from one perspective. Furthermore, the effective lifetime of the technology is uncertain because we do not know when improvements in diagnosis and treatment will come about and how they will influence decision uncertainty.
A few limitations pertain specifically to our study. We applied Dutch medical and nonmedical costs to the entire European Union. This may have biased our results. Moreover, we assumed that medical care in the entire European Union was similar to that in the Netherlands. In addition, in our analyses we assumed that the intervention has no effect on costs and effects after the 6-month follow-up period. Although these limitations may affect the precise figure that results from the calculations, they are unlikely to have a substantial effect on our conclusions.
Our results imply that a Dutch funding agency seeking to maximize future health in the Netherlands should not fund more research regarding the value of MR imaging in patients with acute knee trauma. A European agency, however, should consider funding a multicenter trial with about 3500 patients in each trial arm in which the friction costs, the cost of an overnight hospital stay, and the number of quality-adjusted life-years are measured. However, other unrelated research proposals with a higher expected benefit should receive priority.
Value-of-information analysis is an analytic tool that can help researchers decide whether more clinical research regarding an uncertain medical decision is justified. It is a logical initial step when clinical research is considered regarding a medical decision or when the results of a randomized clinical trial are inconclusive. Value-of-information analysis can be used to determine whether the decision uncertainty justifies the cost of research. Decision uncertainty can be modeled by using all available evidence in the literature (5). Alternatively, the results of a previous clinical study or meta-analysis can be used for value-of-information analysis, as in the current study. Ideally, the analysis should involve all competing strategies to include all decision uncertainty. If the expected benefit of more research is substantial, value-of-information analysis can be used to identify key parameters, evaluate various study designs, and estimate optimal sample sizes. Claxton et al (15,16) demonstrated the feasibility of value-of-information analysis to help guide the research priority setting of the National Health Service in the United Kingdom. Although the mathematics are relatively simple, we realize that it takes time to understand the concepts of value-of-information analysis. To our knowledge, this is the only method with a theoretically sound basis; therefore, we foresee an important role for value of information analysis in guiding future research. The budget for clinical research is limited, and money should be spent where the expected benefits are greatest. Moreover, more clinical research is justified only if the expected benefit of more research exceeds the expected research costs.
|
ADVANCE IN KNOWLEDGE |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
|
IMPLICATION FOR PATIENT CARE |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
|
FOOTNOTES |
|---|
Abbreviations: ENBS = expected net benefit of sampling • EVPI = expected value of perfect information • EVSI = expected value of sample information
2 Current address: Department of Surgery, Amsterdam Medical Center, Amsterdam, the Netherlands. 
Author contributions: Guarantors of integrity of entire study, B.G.K., M.G.M.H.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, B.G.K.; clinical studies, all authors; statistical analysis, B.G.K., T.S., M.G.M.H.; and manuscript editing, B.G.K., E.H.O., M.G.M.H.
Authors stated no financial relationship to disclose.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONADVANCE IN KNOWLEDGEIMPLICATION FOR PATIENT CAREReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  C. Hassan, P. J. Pickhardt, A. Laghi, A. Zullo, D. H. Kim, F. Iafrate, L. Di Giulio, and S. Morini Impact of Whole-Body CT Screening on the Cost-effectiveness of CT Colonography Radiology, April 1, 2009; 251(1): 156 - 165. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
