DOI: 10.1148/radiol.2481072190
(Radiology 2008;248:12-15.)
© RSNA, 2008
When Is the Right Time to Conduct a Clinical Trial of a Diagnostic Imaging Technology?1
Bruce J. Hillman, MD and
Constantine A. Gatsonis, PhD
1 From the Department of Radiology, the University of Virginia Health System, PO Box 800170, Charlottesville, VA 22908 (B.J.H.); and Center for Statistical Sciences, Program in Public Health, Brown University, Providence, RI (C.A.G.). Received December 21, 2007; final version accepted February 21, 2008.
Address correspondence to B.J.H. (e-mail: bjh8a{at}virginia.edu).
Discuss this article at www.rsna.org/radiology/discuss
When is the right time to conduct a clinical trial of a diagnostic imaging technology? It is an important question. The practice of radiology has grown in both scientific and economic influence almost solely because of the dramatic technologic advancements of the past 4 decades. The continuation of the innovation-evaluation-dissemination process is essential to the future robustness of the specialty. In this continuum, evaluation—the process of technology assessment—is playing an increasingly important role as employers and payers demand more definitive evidence of the value of a technology in improving care before granting reimbursement. Without reimbursement, a technology will not be successful.
Unfortunately, there is no simple answer to the question, "when is the best time?" The answer depends on a multiplicity of factors, including the nature of the technology, its intended application, the stage of development and diffusion into practice, from whose perspective the technology is being viewed, and what we will broadly refer to as nonscientific issues. This editorial will address these influences in an effort to provide readers with a better understanding of the thinking involved in clinical trial decision making.
The first step is to define what is meant by a clinical trial of a diagnostic imaging technology. For purposes of this presentation, an imaging clinical trial is any research involving the use of a diagnostic imaging technology for a clinical application on living humans. Thus, we will not consider, for example, the imaging of animals or human specimens or the preclinical testing of devices for their spatial, contrast, or temporal resolution. What we will consider is the evaluation of devices, biologic agents, and pharmaceuticals for the three major applications of diagnostic imaging to clinical practice: screening and early detection of disease, diagnosis and staging of disease, and employment of imaging as a prognostic and/or predictive marker of treatment response.
 |
THE DICTATES OF TECHNOLOGY
|
|---|
It is useful to think about the stages of technology development in determining whether to conduct a clinical trial and what should be the trial end points. Here, the nature of the technology is a prime consideration. For the development of diagnostic contrast media and targeted diagnostic agents (eg, radionuclides), the model developed for the evaluation of therapeutic drugs is applicable and leads to a well-defined, sequenced approach dictated by the stage of their development and is structured in four phases (1). Phase I studies are used to determine whether the agent is safe for use in humans and to determine a range of acceptable doses. Phase II trials usually are an attempt to provide an early estimate of the diagnostic accuracy achieved by using the agent. The goal of a phase II trial is to provide the developers of the agent with sufficient information to determine whether a larger, often much more expensive, clinical trial is warranted. Phase III trials, sometimes called "registration trials" because they are designed to meet the demands of regulatory approval by the Food and Drug Administration (FDA), are multicenter trials in which the diagnostic accuracy achieved with the agent is compared with the accuracy of alternatives, most often imaging conducted with the same modality but without the use of the agent. When phase IV trials are conducted, they usually are intended to provide continuing surveillance of the safety of an agent after marketing or to add evidence of effectiveness that may be used to expand indications for use or for marketing efforts.
The situation is more complex with imaging devices. There is no universally established phased approach. In addition, the level of evidence demanded by the FDA for approval of most new devices tends to be lower than it is for new biologics and pharmaceuticals. As a result, the assessment of new imaging devices tends to be more diffuse and haphazard than it is for new molecules.
Fryback and Thornbury (2) propounded a hierarchical scheme for considering the evaluation of new medical devices in ascending order of meaningfulness to the value of the technology to clinical care. This scheme follows: diagnostic accuracy (accuracy end points such as sensitivity, specificity, and area under the receiver operating characteristic curve), diagnostic and therapeutic thinking efficacy (discerning the impact on referring physicians employing end points that are often based on surveys of physicians' considerations), and health outcomes and costs (example end points include mortality, changes in quality of life, and cost per quality-adjusted life-years). Gatsonis (3) proposed an approach to the clinical evaluation of diagnostic modalities that parallels the phase I–IV study system for therapy. The approach can be presented in a matrix format (Table), with columns corresponding to the "developmental age" of the modality and rows corresponding to different aspects of the "value" of the modality. The developmental age of the modality can be categorized as one of four phases, with technology assessment targets as follows: phase I (discovery), establishment of technical parameters and diagnostic criteria; phase II (introductory), early quantification of performance in clinical settings; phase III (mature), comparison with other modalities in large, prospective, multi-institutional studies (efficacy); and phase IV (disseminated), assessment of the procedure as utilized in the community at large (effectiveness).
The value of the modality is manifested in several domains, including the following: (a) diagnostic and predictive performance (depending on the technology, this may be best represented by its accuracy in lesion detection, lesion characterization, or capability to help predict response to therapy and long-term patient outcomes), (b) impact on process of care, and (c) impact on patient outcomes and costs.
Most radiology research and publication focuses on the lower rungs of these hierarchies, where small studies can be carried out at single institutions and paired designs requiring smaller sample sizes can be employed. Trials that address the impact of imaging modalities on patient outcomes and cost usually require randomization; larger sample sizes; much greater expense; and multi-institutional, multidisciplinary coordination to enforce the rigor required.
These factors have been daunting barriers to conducting more advanced trials. Nonetheless, while it makes little practical sense to conduct a trial to determine the impact on patient outcomes for a new technology that is still developing and changing very rapidly and for which the actual benefits and costs of use in general practice are not known, once the accuracy of a test is established, we should probably be better than we have been as a specialty about moving on to developing trials that better determine what society would really like to know about our innovations—the extent to which employing imaging contributes to improving health and at what cost. To some extent, the methodological, logistic, and financial barriers to pursuing phase III clinical trials for suitable technologies have been overcome by the development of a federally funded imaging clinical trials cooperative group, the American College of Radiology Imaging Network (ACRIN). However, even here there is a limitation of what can be done with the available funding and, because of the National Cancer Institute (NCI) funding source, ACRIN has so far only addressed imaging as it applies to cancer (4).
|
PERSPECTIVE MATTERS
|
|---|
One critical consideration as to whether and when a clinical trial should be performed is from whose perspective the question is being asked. For most imaging technologies, there are numerous points of view: commercial, provider, payer, and regulatory. For any given set of circumstances, some of these matter more than others. As an example, consider a small biotechnology firm that has been developing a new targeted molecular imaging agent. It is living on venture capital. The holders of the intellectual property (the inventors) are watching their share of ownership decline as they try to keep afloat. From their perspective, the time to do a trial is inevitably "as soon as possible"—that is, when they can get past preclinical testing and safely move on to humans. Although the situation is less dramatic for large, established companies, the impetus is the same. Every day of development means more cost and lost revenue, so they too usually wish to conduct their trials at the earliest possible time.
Although payers often claim that their interest in making decisions to reimburse for a technology is related solely to the credibility of evidence supporting clinical use, payers clearly have financial incentives to stave off paying for expensive imaging technologies that may have broad and frequent applications. One would think that payers, having such an important financial stake, would be eager to support and participate in designing and implementing rigorous trials that would definitively inform their reimbursement decision making. In fact, the opposite has usually been the case. Payers mostly maintain that physicians (interestingly, not patients) are the beneficiaries of new technology and should underwrite the cost of evaluating innovations. In recent years, payers have begun to insist that a technology show the capacity to improve patients' health (5). This bar often is difficult for imaging technologies to cross. Imaging tends to be a node in a chain of diagnostic and therapeutic encounters, the sum of which is expected to positively affect outcome. It is often difficult to parse out the role imaging plays in achieving a successful result.
One would think, then, that, because radiologists would benefit intellectually and financially from the advent of and payment for a new imaging technology, our incentives to conduct a trial would be congruent with those of manufacturers—the earlier the better. However, there are pitfalls in this conceit. Aside from the barriers to conducting rigorous, generalizable research already discussed, there may be concern that a negative result from too early a trial may doom an otherwise valuable technology, or at the very least, set it back some years. It is quite a quandary in today's payment environment. Experience has shown that traditional single-institutional radiology research is likely to portray a new technology more positively than it would ultimately appear in multicenter studies or than it would work in general community practice. The generalizability of the results from the small, typically single, institutional studies has always been limited, and such studies are no longer sufficient to promote reimbursement for a modality. Positive results from more rigorous, more generalizable multi-institutional trials are more likely to hasten positive payment decisions. However, such a result is less likely until a technology has sufficiently matured. In addition, because medical imaging already is highly effective, to a very real extent, each new technology faces a stiffer task in demonstrating its superiority than the last.
|
APPROACHES TO TRIALS DECISION MAKING
|
|---|
The argument has famously been made that the first use of a new medical technology should be entered into a clinical trial and that each successive case should be entered as well (6). What a wonderful world it would be if this actually occurred. We would have more reliable data and learn more and sooner about our new technologies. We would progressively ask more sophisticated questions as the technology developed. There would be less wasted use, and the data to support reimbursement decisions would accrue at a dramatically accelerated rate. Unfortunately, there is neither the will nor the logistics to make the early initiation of formal clinical trials happen. In diagnostic imaging, few academic practices have the discipline to participate, let alone early adopters in community practice, who, with rare exceptions, do not recognize clinical research as a mission.
There also is a need for guidance. There are just too many new technologies and novel applications of existing technologies to submit all of them to clinical trials. As a result, methods have been developed to assess the "value of information" that would be obtained from conducting a formal clinical trial and when might be the most propitious time to do so. In an often-cited article, Phelps and Mushlin (7) developed a decision theoretic framework for determining the expected value of diagnostic information. They proposed a two-step approach to decisions about studies of new diagnostic imaging technology. In the first step ("hurdle 1"), the expected value of perfect information is estimated. If this value is less than the cost of using the new technology, any further testing of the technology does not seem justifiable. If a modality overcomes the first hurdle, clinical studies of the diagnostic accuracy of the modality would be performed and used to estimate the expected value of the (imperfect) information that will be provided by the modality in practice. The second step ("hurdle 2") would be overcome if the expected value of the imperfect information gleaned from employing the technology exceeds its cost.
The decision theoretic model of Phelps and Mushlin (7) can also be used to decide on whether to conduct comparative studies of modalities. Specifically, if the receiver operating characteristic curve of the conventionally employed diagnostic modality is available, a "challenge region" can be computed that shows the range of sensitivity and specificity pairs that would be required for the new modality to be sufficiently cost effective in order to represent an improvement. If the challenge region consists of values of sensitivity and specificity that seem to be out of the range of possibilities for the new modality, then a comparative study may not be justifiable.
These decision analytic approaches to the question of what trials should be conducted are conceptually attractive but rarely implemented in practice. Nonetheless, their logic is valuable in thinking about imaging evaluation even if the formal modeling analysis is not performed.
|
HOW DECISIONS TO CONDUCT CLINICAL TRIALS ARE MADE
|
|---|
Kahan et al (8) investigated the factors that influenced whether to conduct a clinical trial for the National Institutes of Health. The authors noted that different considerations become relevant depending on the stage of development of the technology. Early on (what the authors refer to as the "awareness stage"), the prime concerns relate to aspects of the technology and the target disease. Later (the "relevance stage"), political considerations, whether there is a mandate for a trial, and the degree to which there are competitive technologies take precedence. Finally, in the "feasibility stage," decision makers need to determine what methodological constraints might exist, whether an ethical design is feasible, and whether there is a mandate on the part of physicians and patients that will support a successful accrual.
Many of these influences are exemplified in the deliberations leading up to the conduct of the Digital Mammographic Imaging Screening Trial of ACRIN (9). A more complete description of this example of how scientific, technical, methodological, and regulatory influences impinge on clinical trial decision making is available at http://radiology.rsnajnls.org/cgi/content/full/248/1/12/DC1.
The FDA made an early determination that full-field digital mammography posed a sufficient risk of increased false-positive diagnoses beyond what was experienced in screen-film mammography that it posed the threat of increasing the negative biopsy rate (Susan Alpert, PhD, MD, FDA, director of the Office of Device Evaluation, to Morgan Nields, written communication, February 9, 1999). A 1996 FDA Guidance required the submission of data showing the equivalence of full-field digital mammographic and screen-film mammographic interpretations for companies to market their full-field digital mammographic devices. However, the FDA failed a 1999 filing by GE Healthcare, Milwaukee, Wis, by using this approach and then revoked the 1996 Guidance and upgraded full-field digital mammography to class III. They further recommended that future filings address the accuracy of full-field digital mammographic interpretations. This action imposed a higher barrier of proof of the clinical efficacy of full-field digital mammography, as well as greater expense on the companies developing the technologies. In a 1999 letter to Daniel Sullivan, the associate director of the Diagnostic Imaging Program (now the Cancer Imaging Program), at the NCI, Robert Brittain of the National Electrical Manufacturers Association expressed the concern that the manufacturers were already overinvested and that the type of trial being demanded by the FDA was too expensive for any single manufacturer.
There developed a coordinated advocacy effort on behalf of digital mammography before Congress and the NCI. The resulting pressure on NCI was telling. Members of the NCI called together the key stakeholders, including the imaging device companies, the FDA, the Centers for Medicare and Medicaid Services, and the principal investigator of a proposed ACRIN trial of full-field digital mammographic screening, Etta Pisano, MD. That ACRIN existed provided the scientific, logistic, informatics, and accrual capacity to make feasible a trial of the size and level of sophistication demanded by the FDA. NCI Director Richard Klausner, MD, supported the concept for a trial that emerged from the stakeholders' meeting and eventually provided $27 million in general grant funds to ACRIN to pursue the trial. The Digital Mammographic Imaging Screening Trial opened for accrual in 2000, accrued nearly 50 000 women on schedule, and published its primary results that younger and perimenopausal women and women with dense breasts benefit from increased accuracy of interpretations attributable to full-field digital mammography in 2005 (10).
The foregoing example demonstrates that there are critical issues beyond the virtues of the technology and the importance of its application that must be accommodated if a trial is to go forward and when that occurs. The following factors all have an important impact on whether a clinical trial proceeds: the availability of a funding mechanism and research infrastructure; the relative effectiveness of proponents of a trial; the concerns of manufacturers, providers, advocacy groups, and patients; the perceived potential of an innovation to improve health; and political pressures (11).
|
CONCLUSION
|
|---|
In contrast to the evaluation of therapy, the formation of a paradigm for diagnostic imaging evaluation is still a work in progress. Studies of imaging modalities are needed at each stage of their technologic development. Multicenter, typically comparative trials are increasingly needed for regulatory and reimbursement decisions. Decision analytic models can be used to help guide the process of determining the timing and structure of trials of diagnostic accuracy. End points beyond accuracy are essential to proving the value of diagnostic imaging technologies. However, in many situations it will not be feasible to conduct trials to assess the impact of a diagnostic modality on patient outcomes. Decision analysis and modeling provide practical alternatives in such cases.
Discuss this article at www.rsna.org/radiology/discuss
|
FOOTNOTES
|
|---|
B.J.H. was chair of the National Cancer Institute–funded clinical trials cooperative group, the American College of Radiology Imaging Network, which conducts clinical trials of imaging technologies, and currently is chief scientific officer of ACR Image Metrix, an imaging contract research organization that provides clinical trials services to industry. C.A.G. is the network statistician for American College of Radiology Imaging Network.
|
References
|
|---|
- U.S. Food and Drug Administration, Center for Drug Evaluation and Research. Guidance documents. http://www.fda.gov/cder/guidance/index.htm. Accessed December 18, 2007.
- Fryback DG, Thornbury JR. Efficacy of diagnostic imaging. Med Decis Making 1991;11:88–94.[Abstract/Free Full Text]
- Gatsonis C. Design of evaluations of imaging technologies: development of a paradigm. Acad Radiol 2000;7:681–683.[CrossRef][Medline]
- Hillman BJ, Gatsonis C, Schnall MD. The American College of Radiology Imaging Network (ACRIN): a retrospective on 5 years of conducting multi-center trials of radiology and plans for the future. J Am Coll Radiol 2004;1:346–350.[CrossRef][Medline]
- Ziegler KM, Flamm CR, Aronson N. The Blue Cross Blue Shield Technology Evaluation Center: how we evaluate radiology technologies. J Am Coll Radiol 2005;2:33–38.[CrossRef][Medline]
- Chalmers TC. The clinical trial. Milbank Mem Fund Q Health Soc 1981;59:324–329.[CrossRef][Medline]
- Phelps CE, Mushlin AI. Focusing technology assessment using medical decision theory. Med Decis Making 1988;8:279–289.[Abstract/Free Full Text]
- Kahan JP, Neu CR, Hammons GT, Hillman BJ. The decision to initiate clinical trials of current medical practices. Publication no. R-3289-NCHSR. Santa Monica, Calif: Rand Corporation, 1985.
- Pisano ED, Gatsonis CA, Yaffe MJ, et al. American College of Radiology Imaging Network Digital Mammographic Imaging Screening Trial: objectives and methodology. Radiology 2005;236:404–412.[Abstract/Free Full Text]
- Pisano ED, Gatsonis C, Hendrick E, et al. Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 2005;353:1773–1783.[Abstract/Free Full Text]
- Hillman BJ, Kahan JP. Initiating clinical trials: a case study of breast cancer treatment. Publication no. N-2320/3-NCHSR. Santa Monica, Calif: Rand Corporation, 1985.
TOP
THE DICTATES OF TECHNOLOGY
PERSPECTIVE MATTERS
