HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Search for Related Content PubMed PubMed Citation

Evidence-based Radiology: A New Approach to the Practice of Radiology¹

¹ From the Department of Radiology, McMaster University, 1200 Main St W, Hamilton, Ontario, Canada L8N 3Z5 (Harald O. Stolberg, MD, FRCPC, FACR). The members of the Evidence-based Radiology Working Group and their affiliations are listed at the end of this article. Received August 30, 2000; revision requested September 29; revision received November 28; accepted December 22. Address correspondence to H.O.S. (e-mail: stolberg@mcmaster.ca).

	ABSTRACT

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
In this review, the principles of evidence-based health care and their application to radiology are discussed. Evidence-based health care involves the more formal integration of the best research evidence with clinical expertise and explicit acknowledgment of patient values in clinical decision making, as compared with conventional practice. Recently, many health care disciplines have adopted the principles and practice of evidence-based health care. In radiology, including its diagnostic and interventional aspects, these developments have received limited attention. This review of evidence-based health care could, therefore, be useful to radiologists at any stage of their training or career, to encourage the practice of evidence-based radiology. The development of evidence-based health care is described, and evidence-based health care and evidence-based radiology are defined. The importance of evidence-based health care as a new approach to the practice of medicine and its importance for transdisciplinary collaboration are discussed. The skills required to practice evidence-based radiology are identified, and the roles of evidence-based radiology in radiologic practice, education, and research are discussed.

Index terms: Radiology and radiologists • Radiology and radiologists, outcomes studies • Radiology and radiologists, research • Review

	INTRODUCTION

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 

Good doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough. Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of patients (1).

The complexity of radiology today is the result of the introduction of many revolutionary imaging and interventional technologies during the past 30 years. The explosion of medical technology has made it almost impossible for radiologists to assimilate all the information that they need to render high-quality cost-effective care. The challenge is to keep up to date with the rapidly expanding body of medical knowledge and to learn to access, interpret, and apply this knowledge appropriately (2). For radiologists, limited appraisal skills of the scientific literature can be a substantial problem, and there may be limited training in how to integrate research evidence with other types of information, including values, preferences, and circumstances, into what constitutes valid information (1–4). Other barriers include lack of time, rapidly out-of-date textbooks, and disorganized journals, as well as limited access to and suboptimal use of electronic databases.

During periods of change, people value evidence on which to anchor their decisions and validate their choices. The development of the concept of evidence-based health care (EBHC) occurred because of increasing awareness of the limitations of traditional determinants. The principles of EBHC offer a potential solution to problems that have long been encountered, such as requirements for valid up-to-date information for clinical and policy decisions. Evidence-based decision making can be defined as the systematic application of the best evidence to evaluate the available options and decision making in clinical management and policy settings. In this review we discuss the principles of EBHC and their application to radiology.

Ideally, radiologists should have available strategies that will provide current, valid, and relevant information to guide them in their clinical, research, and administrative decisions. This information should be available at the right time, in the right place, and in the right format. At the very least, these strategies will require several components. These components include unbiased syntheses of the current best information, state-of-the-art information technology, and initiatives to optimize the skills with which people use technology and health information. The environment where radiologists make their decisions is changing with the growing integration of picture archiving and communication systems, or PACS, and the computer-based integrated health care enterprise, or IHE (6). In the future, these changes can be used to ensure that there is optimal interaction between the best available information and the values, preferences, and circumstances or context in which the decisions are made (7).

As medical practice becomes busier and the time for reading and reflection becomes ever more precious, the ability to peruse the literature effectively and to become familiar with modern communication systems becomes an essential skill for all physicians (8). EBHC provides some of the methods, principles, and tools that should help busy radiologists meet this challenge.

	OVERVIEW OF EBHC

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
The History of EBHC
"EBHC deals directly with the uncertainties of medicine and has the potential to transform the education and practice of the next generation of physicians" (2). The philosophic origins of EBHC extend back for a long time and are based on the simultaneous development of new information and tools that allow the more rapid and effective dissemination and exchange of that information (9). The power of modern Western medicine derives in great part from its close alliance with the world of science with use of the scientific method to distinguish what is useful from what is not; William Osler said, "The philosophies of one age have become the absurdities of the next" (10,11). For example, Pierre Louis, in postrevolution Paris, rejected the announcement of authorities that venesection was "good for cholera" and provided empiric evidence that commonly accepted practices such as bloodletting had no therapeutic value (10,12,13).

In the current era, the evidence-based approach was refined, consolidated, and named evidence-based medicine by a group led by Dr Gordon Guyatt at McMaster University in Hamilton, Ontario, Canada (13,14). Other terms have been proposed. Evidence-based decision making refers to the process itself, which is designed to help decision makers access and consider the best available knowledge from research to guide their decisions about patient care, new research activities, or the organization of health services. Evidence-based practice is used by the U.S. Agency for Health Care Research and Quality to designate a series of centers in North America that have been charged with producing evidence reports and technology assessments to support guideline development by other groups (15). EBHC is a more encompassing and preferable term. Recently, many health care disciplines have accepted EBHC principles and practice (13).

The Definition of EBHC
EBHC (often referred to as evidence-based medicine) is the process of systematically finding, appraising, and using contemporary research findings as a basis for clinical and policy decisions. It can be practiced in any situation where there is doubt about aspects of clinical diagnosis, intervention, or management. EBHC represents an evolution in the methodology and tools that are used to practice scientific medicine (2,16,17). The practice of EBHC integrates clinical expertise with the best available external clinical evidence from research. In radiology, individual clinical expertise refers to the proficiency and judgment that individual radiologists acquire with education and practice.

Until recently, terms such as efficacy, effectiveness, and efficiency were not part of the language of busy radiologists. EBHC can facilitate the use of these terms. Definitions from the technology assessment literature include efficacy defined as "the probability of benefit to individuals in a defined population from a medical technology applied for a given problem under ideal conditions," whereas effectiveness "reflects performance of a medical technology under ordinary, rather than ideal conditions" (18). Most radiologists work at the effectiveness level. Efficiency studies address the cost-effectiveness of procedures. Wherever one is along the spectrum of efficacy, effectiveness, or efficiency, EBHC principles provide valuable methods to help answer about the relative benefit and harm of new and existing procedures and the relative merits of diagnostic tests.

EBHC and Radiology
In an excellent Perspectives article published in Radiology in 1999 entitled "What’s the Evidence" (17), Beverly P. Wood, MD, MSc, pointed out that "initiatives in evidence-based medicine are developing in medical schools and training programs throughout North America and Europe." To date, these developments have received limited attention in radiology; our review of EBHC should be useful for radiologists at any stage of their training or career, to encourage the practice of evidence-based radiology (EBR). The roles and competencies of specialist physicians include those of medical expert, communicator, collaborator, manager, health advocate, scholar, and professional (19). The role of medical expert requires the development of certain knowledge and skills that were not part of routine postgraduate training in the past. It is important for radiologists to realize that EBR offers solutions that can be applied at many levels of professional involvement. A unique feature of EBR is that it can be used readily by practicing radiologists working at the effectiveness level: performance in their own departments under ordinary, rather than ideal, conditions.

Principles of EBHC
Evidence-based practice differs from conventional practice in two important ways. First, the core of the EBHC methodology is explicitness. Literature is classified according to type (eg, diagnosis, therapy, reviews, guidelines). Explicit criteria are used to render an objective critical assessment. Standard questions are used to assess the methods and establish the validity of the evidence (20). Standard calculations are then used to assess the strength of the results. This explicitness gives the results of EBHC evaluations transparency and reproducibility. Second, the aim of EBHC is to provide practitioners with the rules and tools to perform their own evaluations and consider them as part of their practice (21–23). Practitioners are thus empowered to confidently develop solutions for individual patient and departmental problems, taking into account the best current evidence from research. This is quite different from the process that has been followed in the development of most guidelines in radiology. Such guidelines are dependent on the opinion of "experts," go rapidly out of date, and may not be applicable to an individual practice.

The steps required to practice EBHC have been laid out in many publications (1,3). The first step is to convert the need for information into a specific question to facilitate subsequent literature searches. The research evidence is then tracked down and located by means of systematic searches. The evidence is critically appraised for its validity, impact, and applicability. The results are integrated with other types of information, values, and preferences and the circumstances in which the decision is made. Beyond this, physicians should evaluate the effectiveness, efficiency and the outcome of the decision, and one’s performance and store the appraised evidence for future use.

Step 1: setting the question.—Initially the clinician acquires data regarding the patient and then generates hypotheses to explain the data. Weinstein and Fineberg (24) described this process as clinical decision analysis. It involves hypothesis formulation and supported forward reasoning. A good hypothesis minimizes ambiguity by distilling the essence of a problem into a single statement. The question that initiates the evidence-based search can relate to diagnosis, treatment, or prognosis. It can involve outcomes of interest, as well as the benefits and harms of interventions and the quality of care (2).

Learning to formulate the question is a fundamental skill for practicing EBHC. The "anatomy" of the question should include four parts: the problem being addressed, the intervention being considered, the comparison of the intervention, and the outcome of interest. The success of the search will depend on how the question is formulated. The hypothesis must not only be internally rigorous but must also be clinically relevant, so that proof or disproof of the hypothesis influences diagnostic or therapeutic decisions. Once a clear diagnostic hypothesis has been formulated, an efficient search of the literature can then be conducted. The alternative is an unstructured search for evidence that can increase the risk of reliance on a partial piece of reality.

Step 2: finding the evidence.—Learning by means of inquiry is dependent on the ability to find the current best evidence to manage clinical and organizational problems. This is a task that can either be quick and highly rewarding or time-consuming and frustrating.

Until recently, the problems of locating the appropriate evidence were often insurmountable for the practicing radiologist, because of the lack of effective searching skills and easy access to databases. Two types of electronic databases are available: The first is bibliographic and permits the user to identify relevant citations from the literature by using variations of MEDLINE. The second takes users directly to quality- and relevance-filtered publications. International groups such as the Cochrane Collaboration are closing the knowledge gap through efforts at systematic review of the literature and publication of the highest quality research based on selected methodological criteria and facilitation of the reader’s understanding of the quality and applicability of such results to their practice. Neither the Cochrane Collaboration nor the available secondary literature include much information about radiology to date. Their development will be important if the current best evidence is to become accessible to the radiology community in general (2,25). A number of commercial groups already offer abstracts, such as Current Contents or the "alert" services provided by many scientific journals. When one subscribes to these, one provides key words and receives in return a notice as soon as a relevant article appears in print. No doubt these services will be refined and will serve to facilitate access to "distilled" high-quality information.

It has also been postulated that if evidence-based decision making is to reach its full potential and contribute to improvements in health care, a powerful and efficient synergy must be developed between it and the Internet. The Internet could benefit evidence-based decision making by giving decision makers inexpensive, fast, and efficient access to up-to-date, valid, and relevant knowledge at the right time, at the right place, and in the right format. Conversely, the tools and principles of evidence-based medicine could be used to gain a better understanding of the role of the Internet in health care, helping us to anticipate opportunities and prevent potential problems (26).

Step 3: critical appraisal.—The next step is to evaluate or appraise the evidence for its validity and usefulness. This step is crucial because it lets the radiologist decide whether an article can be relied on for useful guidance. Unfortunately, a large portion of the published radiology research lacks either relevance or sufficient methodologic rigor to be reliable enough to help answer such questions. A structured but simple method should enable individuals without research expertise to evaluate radiology publications. Critical appraisal skills entail learning how to evaluate the validity of evidence and its relevance to a particular patient or group of patients. The fundamentals can be learned in a few hours with small tutorials, workshops, and interactive lectures (2). Resources for critical appraisal are listed in the Appendix.

Step 4: developing solutions for individual patients.—The final step integrates the results of the appraisal with radiologic expertise and the patient’s unique biologic characteristics, values, and circumstances (27). External evidence can inform but never replace individual radiologic expertise. It is this expertise that is used to decide whether the external evidence applies to the individual patient at all and, if so, how it should be integrated into a radiologic decision (5).

Several recent developments have made the practice of EBHC easier. These include strategies for efficient location and appraisal of the evidence and for decisions on the validity of the published evidence with regard to diagnosis, prognosis, harm, therapy, and outcomes (28–37).

	THE PRACTICE OF EBR

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
Contemporary radiology includes both diagnostic imaging and interventional radiology. The principles of EBHC are equally applicable to both. The recognition of the need for evidence-based decision making comes at a time when information science is available to physicians not just in the library or study but at the point of care. As it becomes increasingly possible to provide real-time decision support for medical decisions, the incorporation of the best evidence into this support has the potential to translate the evidence directly into care. EBR, then, is a way to solve clinical problems by quickly identifying the best available research to guide and support clinical decisions. The practice of EBR empowers radiologists to find solutions for individual patients’ problems, as well as to develop local departmental policies. Unless evidence-based practice improves care, however, it remains no more than a sterile academic exercise.

EBR is aimed at the integration of evaluative sciences and technology assessment into clinical practice (38). Radiologists can apply the EBR rules and tools to original research, reviews published in the literature, and local audits. The results of relevant research must include both estimates of the extent to which study results can be generalized from the study sample as a whole and recognition of the importance of local prevalence of disease on the interpretation of positive and negative test results (confidence intervals and predictive values corrected for local prevalence). To start with, there are many relevant contributions developed by radiologists to address these issues (39–49).

Evidence-based methodology is not required for many diagnostic situations, but it can provide the answers to questions that are common, important, and a problem. It is also most helpful when a personal "knowledge gap" or a rare condition is encountered. In the past, the practice was to follow the example of experts (25). This is a quick model of practice and quite suitable for conditions that are seen rarely and for which ongoing advice is needed. Practice in this mode, however, does not allow the determination of whether the advice received is authoritative or authoritarian.

There are two modes for radiologists to achieve evidence-based practice: The first mode, searching and appraising, encompasses all four steps of EBHC and is demanding in terms of both time and resources. It is most suitable for conditions that are encountered regularly. The second mode encompasses only steps 1, 2, and 4 and limits the searches to sources that have undergone critical appraisal by others. This is a quick approach and is most appropriate for conditions that are seen infrequently. Teaching hospitals, however, should practice and teach all steps and encourage the development of their own critical appraisal.

EBR and Diagnostic Imaging
Given the increasing number and complexity of diagnostic imaging methods, clinicians are more frequently asking radiologists for advice on the appropriate selection of radiologic investigations (50). However, advising on the appropriate use of tests is difficult (51,52). The application of EBR principles to diagnostic imaging can enhance the interpretation of imaging studies and form the foundation for a thorough and meaningful radiologic consultation.

Studies reported in radiology manuscripts do not usually involve advanced statistical analyses (53,54). The majority of the radiology literature concerns technical and diagnostic test performance, which are the first two levels in the technology-assessment hierarchy developed by Thornbury and Fryback (41,55) (Table 1). The hierarchy of efficacy is now the widely accepted foundation for a conceptual approach to technology assessment. A recent attempt has been made to suggest a similarly phased evaluation of new interventional procedures (56). Until recently few authors have investigated diagnostic effect, the third of the Thornbury’s six levels (Table 1); that is, has the radiologic investigation altered the clinician’s diagnosis or rendered other investigations unnecessary? This is important because, in the final analysis, the clinician’s real question is "What is the posttest probability of disease?" or "Given this test result in this patient, has the result altered the disease probability so that it now lies above a treatment (action) threshold, or below an exclusion threshold?" (57). If the answer is yes, the treatment can begin or the suspected condition can be excluded. Otherwise, further testing is needed. It is at this point that EBHC data analysis supplements the traditional technology-assessment analysis, allowing application of a test result observed in a group of patients to an individual patient.

The primary purpose of screening is to prevent or delay death due to disease; hence, the most appropriate outcome measure is disease-specific mortality: the ratio of number of deaths caused by the target disease to the number of person years of observation. Comparisons of disease-specific mortality are not subject to the same biases, because these comparisons are based on populations tracked from the time of screening (or no screening), rather than simply cases tracked from the time of diagnosis. Randomized controlled trials are considered the most valid method for determining the effectiveness of screening for reduction of disease-specific mortality (61). However, randomized controlled trials of screening have several limitations, including problems with compliance among study subjects, lack of generalizability, requirement for a large number of participants, many years of follow-up to produce a statistically significant result, and cost. Because of these limitations, quantitative decision analysis is increasingly used to fill in the gaps of knowledge about the effects of screening (62,63). Decision models can be thought of as virtual trials that can be programmed to analyze the effects on the benefits, harms, and costs of screening of controllable factors (eg, starting or stopping age) and uncontrollable factors (eg, natural history of the disease or competing mortality).

Because screening involves individuals who are asymptomatic, it should not be offered unless the evidence of its effectiveness is especially strong. Furthermore, it has been argued that candidates for screening should provide informed consent and be made aware of their personal risk of disease, the accuracy of the screening tests, the effectiveness of screening in terms of absolute risk reduction, and the potential harms from the screening procedure (64–66).

EBR and Interventional Radiology
Interventional radiology was one of the major advances in the treatment of "surgical" disorders (ie, disorders treated with surgery) of the late 20th century. Radiologists are proud of it, and interventional radiology continues to evolve rapidly. Why should an interventional radiologist bother with EBR? Do the existing academic structures and industry partners not serve interventional radiologists well, providing all the information they need, whether they are involved in developing new techniques or learning how to incorporate them into their busy practices? Interventional radiology is exciting but shares some common problems with regard to interpretation of the literature and listening to experts. When has a new procedure been tested well enough to be introduced into practice? Does a study conclusion show that a new technical option or technique is safe and reliable in practice? Can the literature distributed by industry representatives really be trusted when making a case to start something new in a department? How can it be decided that all sides of the story have been presented or discussed when a refresher course or review article suggests that a new technique is a major advance? Anybody who has spent time (and/or money) working on biliary lithotripsy, laser angioplasty, atherectomy, and early metal stent designs knows these problems only too well.

The answers to these questions can be found by using the EBR approach, which, as described earlier in this article, represents a structured and transparent approach to the formulation of clinical questions and to the identification of the current best evidence about those questions. The EBHC rules and tools for the assessment of benefit and harm can be usefully applied to interventional radiology (56,67).

External pressures from government and insurance payers are now focusing the attention of the interventional radiology community on the necessity of providing interventional radiology research results that can be used to justify (or stop) the expenditure of scarce funds on expensive interventional radiology procedures. It should be possible to construct a hierarchical or phased model to evaluate the efficacy, effectiveness, and efficiency of interventional radiology procedures. A model based on technology assessment and EBHC theory has been suggested by Malone and MacEneaney (56). Rather than representing strict, separate levels and categories of evidence, this model should be regarded as a continuum analogous to that described by Fryback and Thornbury (41,55) (Table 1).

In defense of existing interventional radiology research, it should be said that there is only a small window of opportunity after the introduction of a new technique when ethical committees will allow patients to be randomly assigned to treatments and therefore "deny" half of the patients the new and supposedly better technique (68). By EBHC standards, any deficiency is not predominantly in the quality of the basic research, which is analogous to the safety and efficacy (phase 1) trials in the drug industry; the deficiency lies more in the lack of a general recognition that it is necessary to go beyond the case series format to establish higher levels of evidence before new techniques come into general use. Case-control and cohort studies can provide useful information (because control groups are used) before large, expensive, randomized controlled trials are contemplated.

	EBR AND EDUCATION

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
The contribution of EBHC to teaching and learning provides a firm foundation for the use of the tools of evidence in clinical decision making (17). Inclusion of EBHC in medical education changes the way learners use information, although the core content of medical teaching remains the same. Educators and learners in undergraduate and residency programs are moving away from passive acceptance and toward active searching and analysis of available information. The skills required in EBHC include judgment of the reliability and validity of information, consideration of its application to patients’ problems, and a self-directed approach to education (69).

In traditional medical education, students are taught that the progression of disease and appropriate treatments can be determined through knowledge of the mechanisms of disease. Education in EBHC adds understanding of the rules of evidence that are necessary for effective evaluation and application of findings from the medical literature. Medical students and residents are learning the skills needed to access, select, interpret, and apply appropriate information efficiently and effectively. It is now essential for practicing clinicians to catch up. To acquire the critical thinking skills used in EBHC, physicians must become more self directed in identifying and meeting personal learning needs and committed to lifelong learning (70–72).

The most effective learning occurs when the learner actively seeks, finds, and applies relevant information to a problem that is important and interesting (73,74). These key concepts are the basis for the recommendations for learning and teaching EBHC methodology described by Sackett et al (13). Educational materials in EBHC for all levels of training and practice in radiology must be developed (75,76). For radiologists who are interested in teaching EBR, the second edition of Sackett and associates’ publication, "Evidence-based Medicine: How to Practice and Teach EBM" (13); Users’ Guides to the Medical Literature: A Manual for Evidence-based Clinical Practice (77); and the Health Information Research Unit Web site (78) are excellent starting points.

Education in EBR requires faculty members to become authoritative rather than authoritarian. It also requires an interactive, rather than a didactic, learning environment. Problem-based small group discussions, journal clubs, rounds, and consultations with colleagues are opportunities to enhance the learning of EBR methodology and its application to practice. A commitment of support by department heads, residency directors, and opinion leaders to provide EBHC education in radiology is essential for the continuing viability of radiology as a profession. Hence, radiology departments must encourage faculty to practice EBR, emphasize EBR in residency programs, and make EBR methodology part of the core curriculum for residents. They should also make resources for EBR practice and teaching available to faculty and community radiologists and ensure that all continuing medical education and continuing professional development programs in radiology have EBR content.

	EBR AND RADIOLOGY RESEARCH

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
Research can be viewed from several perspectives, including those of users, researchers, educators, and policy makers. Each has different fundamental needs. Users are the radiologists working at the effectiveness and efficacy levels in community and university hospitals. Ideally, they should use original literature, as well as quality- and relevance-filtered secondary publications. New and updated knowledge is used to design state-of-the-art departmental policies and to interpret individual patients’ examination findings. The key concerns for this group are ease of access to radiologic information and the reliability, reproducibility, and local applicability of the conclusions presented in the literature. The key feature of EBHC in this respect is that EBHC methods were written by physician-statisticians (clinical epidemiologists) for physicians who will appraise the published literature.

Researchers are radiologists and nonradiologists working at the efficacy and efficiency levels of technology assessment. Incentives in academic radiology have been counterproductive, encouraging researchers to publish early and often rather than conduct long-term, large, and time-consuming studies. Access to new technology has generally ensured that early investigators become well known for publishing case series (79). On the other hand, acceptable research methodology requires that it be preplanned, comparative, and hypothesis driven, not simply descriptive. Protocols for conducting studies should include specific details describing every aspect of the study. This is essential for making valid reliable comparisons between studies. The use of empiric evidence to justify medical practice is a powerful principle but difficult to defend. Initiatives such as the American College of Radiology Imaging Network, or ACRIN, will facilitate high-quality primary research studies and online reports and should be supported (80).

When designing or assessing research, both diagnostic imaging and interventional radiology must be considered. The level of evidence is graded from the available information on diagnostic and interventional study design (levels 1–5 in Tables 2 and 3) (20). Validity can be assessed with standard questions designed to reveal systematic bias (81). The strength of the study is established with calculations using study result data. Many of these calculations can be performed by using a spreadsheet approach (20,23,82). In addressing issues of test accuracy in diagnostic imaging, level 1 evidence is provided by an independent blinded comparison of an appropriate spectrum of consecutive patients, all of whom have undergone both the diagnostic test and a reference standard examination. The level of evidence decreases predictably if these standards are breached. The strength of the study may be derived from contingency (two-by-two) table analysis, with confidence intervals used as a measure of imprecision.

Analyses of diagnostic accuracy are often restricted to the calculation of sensitivity, specificity, and the predictive values of positive and negative test results. These analyses provide an overview of test performance in a particular study population and a comparison of test performance against the performance of other tests. Test sensitivity and specificity values are, in the radiology literature, often reported without confidence intervals.

To determine the relevance of a diagnostic test result to an individual patient, an estimate of pretest probability is needed. In populations, pretest probability is equivalent to disease prevalence. It can also be the clinician’s and the radiologist’s estimate of disease probability in an individual patient after taking into account all available data (83). Once pretest probability is known, posttest probability can be calculated by using likelihood ratios (Bayesian theory). Likelihood ratios summarize the information in both sensitivity and specificity, are applicable to tests in which there are more than two levels of possible results, and, with use of a nomogram, provide an efficient way of moving from pretest to posttest probability (57). An alternative to the nomogram is the computer spreadsheet, which may represent the ideal medium for automatic, instantaneous, repetitive calculations based on formulas, which the user of the spreadsheet need never enter (64,82). There are inherent weaknesses in this model of results analysis, since continuous data are treated in a binary fashion. Receiver operating characteristic, or ROC, curves are occasionally calculated. ROC curves offer a more rigorous assessment of diagnostic test performance. In the future, methods will be developed to account for the anatomic extent of disease relevant to imaging (84).

From the educators’ perspective, the challenge is to incorporate current research evidence into a meaningful curriculum for all levels of learning and practice.

Policy makers today face the difficult task of stimulating better research while making good decisions based on the best available evidence. The development of the capacity for evidence-based decision making is achieved not only by individual skills but also by changes in the culture, systems, and structure of organizations (85). When the evidence is strong, subsequent decisions are easy; when it is weak, good judgment is required to make the correct practice and policy decisions (86).

Outcomes research has received a great deal of attention in recent years, but there are substantial barriers to outcomes research that are particular to radiology. These barriers, as well as the traditional health outcomes in the evaluation of diagnostic tests and the diagnostic and therapeutic effects of intermediate outcomes, are important components of EBR and were extensively addressed recently in Academic Radiology (87).

	PERSPECTIVE

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
The rapid spread of EBHC has resulted in the practice and teaching of EBHC in many medical and other health care disciplines. Radiology, in either its diagnostic or its interventional aspects, has so far not participated to any substantial degree in these developments, although there have been concerted efforts by leaders in radiology to promote this approach. To quote Hillman (43), "little of radiological practice is evidence-based" and "the 21st century will need a very different type of radiologist." For these reasons, we advocate the introduction of the principles of EBHC in all aspects of radiology, including practice, education, and research. In doing so, we accept Wood’s statement that the advent of EBHC does not imply that we should overturn our previous methods (17). We also accept Jadad and Enkin’s (9) discussion of the "tacit dimensions of knowledge." EBR requires not only the "best available evidence" but individual radiologic expertise. Radiologists’ expertise is based on their extensive training in the understanding of the structure and function of the human body, the mechanism of disease, the physics principles of image formation, and image interpretation. The tools now in evolution for the practice of scientific radiology are "supported by a growing base of scientifically designed studies for the valid evaluation of diagnostic criteria, interventional procedures and management decisions" (25).

The shortage of coherent and consistent scientific evidence in the radiology literature is a distinct disadvantage for EBR. EBR must, therefore, promote the application of radiologic expertise and reasoning based on principles derived from basic science research to traverse these many gray zones of practice (88). Even when evidence exists, difficulties arise when the evidence is inconclusive, inconsistent with previous results, irrelevant to clinical realities, or of poor quality (89). Although systematic reviews are a potential solution to this problem, inadequate attention to their methods may lead to surprising variations in results and recommendations. Steps must, therefore, be taken to improve and standardize the methods and reporting of systematic reviews (89). These problems, far from constituting a limitation of EBR, however, highlight the importance of training radiologists to appraise research critically and apply the evidence, taking into account their patients’ individual risks and values (89).

Unquestionably, the practice of EBR requires the acquisition and development of new skills in literature searching and critical appraisal. Mastery and application of these skills are tasks that should not be underestimated, although the fundamentals of critical appraisal can be learned in a few hours in small tutorials, workshops, and interactive lectures (89). Even radiologists with limited experience in literature searching can learn how to apply EBR in the context of their own practice. It should also be possible to integrate standard interpretive radiology teaching materials with noninterpretive EBR content during refresher courses (90).

The advantages of practicing EBR, however, outweigh all disadvantages and limitations. Individual radiologists are enabled to routinely upgrade their knowledge base, to improve their understanding of research methods, and to use data in a more critical fashion. Radiologists improve their computer literacy skills, data searching techniques, and reading habits, as well as improve their confidence in management decisions. For radiology teams, EBR provides a framework for group problem solving and for teaching and enables junior members of the team to contribute usefully. For patients, it ensures more effective use of resources and better communication about the rationale behind radiology management decisions. While the contributions of junior members of the team are essential, support by senior radiologists is critical to the success of introducing EBR. Senior radiologists who practice EBR regularly are excellent role models who can train newcomers. Even when senior staff are themselves unfamiliar with EBR, their willingness to admit uncertainty, to encourage skepticism, and to be flexible can help the team to accommodate new evidence that may contradict previous assumptions and practice (2).

The practice of EBR will encourage and facilitate interdisciplinary collaboration (91). A prerequisite to the promotion of high standards of specialist care in radiology is the optimal use of imaging and interventional procedures in daily practice. The principles of EBHC can be applied to all aspects of radiology and will help promote the appropriate use of imaging procedures, enhance the interpretative accuracy of image reading, improve the application of interventional procedures, and form a foundation for a thorough and meaningful radiologic consultation.

	APPENDIX

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 
EBHC Web Sites
McMaster University (Hamilton, Ontario, Canada). Health Information Research Unit: Evidence-Based Health Informatics. Available at: hiru.mcmaster.ca/default.htm. Accessed August 24, 2000.

Oxford University (England). Centre for Evidence-Based Medicine. Available at: cebm .jr2.ox.ac.uk/. Accessed August 24, 2000.

The University of Sheffield (England) School of Health and Related Research. www .nettingtheevidence.org.uk: A ScHARR Introduction to Evidence Based Practice on the Internet. Available at: www.nettingtheevidence .org.uk. Accessed June 18, 2001. This site, by Andrew Booth, is a well-organized and most comprehensive jumping-off point. It leads to all the important sites. Users can "surf" in the direction of their choice at their own pace.

Critical Appraisal with EBHC Principles
Centres for Health Evidence (Edmonton, Alberta, Canada). Principles: User’s Guides to Evidence-Based Practice. Available at: www.cche.net/principles/content_all.asp. Accessed August 24, 2000.

Data Analysis: Diagnostic Radiology
The Meaning of Diagnostic Test Results: A Spreadsheet for Swift Data Analysis. Available at: ftp://radiography.com/pub/Rad-data99.xls. Accessed August 18, 2000.

The Evidence-Based Medicine Toolbox. Available at: cebm.jr2.ox.ac.uk/docs/toolbox.html. Accessed August 24, 2000.

Agency for Healthcare Research and Quality home page. Available at www.ahcpr.gov/. Accessed November 6, 2000.

Data Analysis: Interventional Radiology
Applying "Evidence-Based Medicine" Theory to Interventional Radiology: A Spreadsheet for the Swift Assessment of Procedural Benefit and Harm. Available at: ftp://radiography.com/pub /TxHarm00.xls. Accessed August 24, 2000.

Literature Searching
For primary sources, the free sites used by most people are PubMed (available at: www .ncbi.nlm.nih.gov/Pubmed; accessed August 24, 2000) and Grateful Med (available at: igm.nlm.nih.gov; accessed August 24, 2000). Subscription-based search programs include Ovid (available at: www.ovid.com; accessed August 24, 2000), and Silver Platter (available at: www.silverplatter.com; accessed August 24, 2000). Radiology journals online include the RSNA Index to Imaging Literature (available at: rsnaindex.rsnajnls.org; accessed August 28, 2000), Radiology Online (available at: radiology.rsnajnls.org; accessed August 28, 2000.), and the American Journal of Roentgenology (available at: www.arrs.org/ajr; accessed August 24, 2000).

There is, as yet, little secondary literature for radiologists. Other disciplines have created a wealth of secondary literature. Robin Snowball, BA, of the Cairns Library (Oxford University, England) has categorized information resources as follows: databases (and indexes), journals, bulletins and newsletters, "gateways" and search engines, discussion lists and support groups, guidelines, health economics, health outcomes, evidence-based medicine Web sites, and electronic journals. A list of selected Internet addresses for all these categories has been published (92), with advice on how to get the best results from their searching strategies.

A comprehensive review for EBHC is available in the article by Jadad et al (26) and at the McMaster University Health Information Research Unit Evidence-Based Health Informatics Web site (available at: hiru.mcmaster.ca; accessed June 5, 2001).

	ACKNOWLEDGMENTS

 
There are a large number of leaders in radiology and clinical epidemiology who provided the inspiration for this review. Our special thanks go to Gordon H. Guyatt, MD, FRCP(C), of the Department of Clinical Epidemiology and Biostatistics, McMaster University (Hamilton, Ontario, Canada) for encouragement, help, and a review of this report. We are very grateful to John R. Thornbury, MD, for his help. We all thank Monika Ferrier, BA, of the Department of Radiology, McMaster University, for her tireless efforts in communicating with members of the Working Group and for preparing the manuscript.

	FOOTNOTES

 
Authors and affiliations: William C. Black, MD, Department of Radiology, Dartmouth-Hitchcock Medical Center, Lebanon, NH, and Center for the Evaluative Clinical Sciences, Department of Community and Family Medicine, Dartmouth Medical School, Hanover, NH; Alejandro R. Jadad, MD, DPhil, FRCPC, Departments of Health Administration and Anesthesiology, University of Toronto, Ontario, Canada; Jeffrey G. Jarvik, MD, MPH, Departments of Radiology and Neurosurgery, University of Washington, Seattle; Ella A. Kazerooni, MD, MS, Department of Radiology, University of Michigan, Ann Arbor; Curtis P. Langlotz, MD, PhD, Departments of Radiology, Epidemiology, and Computer and Information Science, University of Pennsylvania Medical Center, Philadelphia; Brian C. Lentle, MD, Department of Radiology, University of British Columbia, Vancouver, Canada; Peter M. MacEneaney, FFR, RCSI, Department of Radiology, University of Chicago Hospitals, Ill; Dermot E. Malone, MD, FRCPI, FFRRCSI, FRCR, FRCPC, Department of Radiology, St Vincent’s University Hospital, Dublin, Ireland; Claude Nahmias, PhD, Department of Nuclear Medicine, McMaster University Medical Centre, Hamilton, Ontario, Canada; Martin H. Reed, MD, Department of Pediatrics and Child Health, Children’s Hospital, Winnipeg, Manitoba, Canada; Bruno J. Salena, MD, FRCPC, Division of Gastroenterology, Susan I. Shannon, PhD, Faculty of Health Sciences, and Harald O. Stolberg, MD, FRCPC, Department of Radiology, McMaster University, Hamilton, Ontario, Canada.

Abbreviations: EBHC = evidence-based health care, EBR = evidence-based radiology

	REFERENCES

TOP ABSTRACT INTRODUCTION OVERVIEW OF EBHC THE PRACTICE OF EBR EBR AND EDUCATION EBR AND RADIOLOGY RESEARCH PERSPECTIVE APPENDIX REFERENCES

 

1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ 1996; 312:71-72.[Free Full Text]
2. Sackett DL, Rosenberg WMC. The need for evidence-based medicine. J R Soc Med 1995; 88:620-624.[Abstract]
3. Rosenberg W, Donald A. Evidence based medicine: an approach to clinical problem-solving. BMJ 1995; 310:1122-1126.[Free Full Text]
4. Jadad AR. Randomised controlled trials: a user’s guide London, England: BMJ Books, 1998; 98.
5. Black D. The limitations of evidence. J R Coll Physicians Lond 1998; 32:23-26.[Medline]
6. Jost RG. The IHE initiative (abstr). Radiology 1999; 213(P):597.
7. Choi PT, Jadad AR. Systematic reviews in anesthesia: should we embrace them or shoot the messenger? (editorial). Can J Anesth 2000; 47:486-493.[Free Full Text]
8. Friedland DJ. Evidence-based medicine: a framework for clinical practice Stamford, Conn: Appleton & Lange, 1998.
9. Jadad AR, Enkin MW. The new alchemy: transmuting information into knowledge in an electronic age. CMAJ 2000; 162:1826-1828.[Free Full Text]
10. Starr P. The social transformation of American medicine New York, NY: Basic Books, 1982.
11. Osler W. Aequanimitas; with other addresses to medical students, nurses and practitioners of medicine Philadelphia, Pa: Blakiston, 1905.
12. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA 1992; 268:2420-2425.[Free Full Text]
13. Sackett DL, Straus SE, Richardson S, Rosenberg WMC, Haynes BR. Evidence based medicine: how to practice and teach EBM 2nd ed. Edinburgh, Scotland: Churchill Livingstone, 2000.
14. Guyatt GH. Evidence-based medicine (editorial). Ann Intern Med 1991; 114(suppl 2):A16.
15. Agency for Healthcare Research and Quality home page.; Available at: www.ahcpr.gov/. Accessed November 6, 2000..
16. Jadad AR, Haynes RB. The Cochrane Collaboration: advances and challenges in improving evidence-based decision making. Med Decis Making 1998; 18:2-9.[Abstract/Free Full Text]
17. Wood BP. What’s the evidence?. Radiology 1999; 213:635-637.[Free Full Text]
18. Brook RH, Lohr K. Efficiency, effectiveness variations and quality: boundary crossing research. Med Care 1985; 23:710-722.[Medline]
19. The Royal College of Physicians and Surgeons of Canada. Canadian medical education directions for Specialists 2000 Project, September 1996.; Available at: rcpsc.medical .org/english/publications/canmed_e.html. Accessed November 6, 2000..
20. NHS Research and Development, Centre for Evidence-Based Medicine. CATbank topics: Levels of Evidence.; Available at: cebm.jr2.ox.ac.uk/eboc/eboclevels.html. Accessed August 18, 2000.
21. Jaeschke R, Guyatt GH, Sackett DL, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients?. JAMA 1994; 271:703-707.[CrossRef][Medline]
22. Jaeschke R, Guyatt GH, Sackett DL, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. III. How to use an article about a diagnostic test. A. Are the results of the study valid?. JAMA 1994; 271:389-391.[CrossRef][Medline]
23. MacEneaney P, Malone DE. The meaning of diagnostic test results: a spreadsheet for swift data analysis. Clin Radiol 2000; 55:227-235.[CrossRef][Medline]
24. Weinstein MC, Fineberg HV. Structuring clinical decisions under uncertainty In: Clinical decision analysis. Philadelphia, Pa: Saunders, 1980; 12-36.
25. Straus SE, McAlister FA. Evidence-based medicine: past, present, and future. Annals of the Royal College of Physicians and Surgeons of Canada 1999; 32:260-264.
26. Jadad AR, Haynes RB, Hunt D, Browman GP. The Internet and evidence-based decision-making: a needed synergy for efficient knowledge management in health care. CMAJ 2000; 162:362-365.[Free Full Text]
27. McAlister FA, Straus SE, Guyatt GH, Haynes RB, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. XX. Integrating research evidence with the care of the individual patient. JAMA 2000; 283:2829-2836.[Abstract/Free Full Text]
28. Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. IV. How to use an article about harm. JAMA 1994; 271:1615-1619.[CrossRef][Medline]
29. Laupacis A, Wells G, Richardson WS, Tugwell P, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. V. How to use an article about prognosis. JAMA 1994; 272:234-237.[CrossRef][Medline]
30. Oxman AD, Cook DJ, Guyatt GH, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. VI. How to use an overview. JAMA 1994; 272:1367-1371.[CrossRef][Medline]
31. Richardson WS, Detsky AS, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid?. JAMA 1995; 273:1292-1295.[CrossRef][Medline]
32. Richardson WS, Detsky AS, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients?. JAMA 1995; 273:1610-1613.[CrossRef][Medline]
33. Naylor CD, Guyatt GH, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. X. How to use an article reporting variations in the outcomes of health services. JAMA 1996; 275:554-558.[CrossRef][Medline]
34. Dans AL, Dans LF, Guyatt GH, Richardson S, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. XIV. How to decide on the applicability of clinical trial results to your patient. JAMA 1998; 279:545-549.[Free Full Text]
35. Richardson WS, Wilson MC, Guyatt GH, Cook DJ, Nishikawa J, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. XV. How to use an article about disease probability for differential diagnosis.. JAMA 1999; 281:1214-1219.[Free Full Text]
36. Barratt A, Irwig L, Glasziou P, et al. for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. XVII. How to use guidelines and recommendations about screening. JAMA 1999; 281:2029-2034.[Free Full Text]
37. Randolph AG, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH, for the Evidence-Based Medicine Working Group. Users’ guides to the medical literature. XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. JAMA 1999; 282:67-74.[Free Full Text]
38. Eisenberg JM. Ten lessons for evidence-based technology assessment. JAMA 1999; 282:1865-1869.[Free Full Text]
39. Begg CB, McNeil BJ. Assessment of radiologic tests: control of bias and other design considerations. Radiology 1988; 167:565-569.[Abstract]
40. Black WC. How to evaluate the radiology literature. AJR Am J Roentgenol 1990; 154:17-22.[Free Full Text]
41. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11:88-94.
42. Hillman BJ. Technology assessment and radiologist researchers (editorial). Invest Radiol 1991; 26:109.[CrossRef][Medline]
43. Hillman BJ. Medical imaging in the 21st century. Lancet 1997; 350:731-733.[CrossRef][Medline]
44. Hillman BJ. Critical thinking: deciding whether to incorporate the recommendations of radiology publications and presentations into practice. AJR Am J Roentgenol 2000; 174:943-946.[Free Full Text]
45. McNeil BJ, Adelstein SJ. Determining the value of diagnostic and screening tests. J Nucl Med 1976; 17:439-448.[Free Full Text]
46. Sunshine JH, McNeil BJ. Rapid method for rigorous assessment of radiologic imaging technologies. Radiology 1997; 202:559-557.[Abstract]
47. Thornbury JR, Eugene W. Caldwell Lecture: clinical efficacy of diagnostic imaging—love it or leave it. AJR Am J Roentgenol 1994; 162:1-8.[Abstract/Free Full Text]
48. Thornbury JR. Why should radiologists be interested in technology assessment and outcomes research?. AJR Am J Roentgenol 1994; 163:1027-1030.[Abstract/Free Full Text]
49. Thornbury JR, Kido DK, Mushlin AI, Phelps CE, Mooney C, Fryback DG. Increasing the scientific quality of clinical efficacy studies of magnetic resonance imaging. Invest Radiol 1991; 26:829-835.[Medline]
50. Chang PJ. Bayesian analysis revisited: a radiologist’s survival guide. AJR Am J Roentgenol 1988; 152:721-727.
51. Armstrong P, Black WC. Optimum utilisation of radiological tests: the radiologist as advisor. Clin Radiol 1989; 40:444-447.[CrossRef][Medline]
52. Dorfman GS. Utilization of diagnostic tests: assessing appropriateness. Acad Radiol 1999; 6(suppl 1):S40-S51.
53. Goldin J, Zhu W, Sayre JW. A review of the statistical analysis used in papers published in Clinical Radiology and British Journal of Radiology.. Clin Radiol 1996; 51:47-50.[CrossRef][Medline]
54. Elster AD. Use of statistical analysis in the AJR and Radiology: frequency, methods and subspeciality differences. AJR Am J Roentgenol 1994; 163:711-715.[Abstract/Free Full Text]
55. Thornbury JR. Intermediate outcomes: diagnostic and therapeutic impact. Acad Radiol 1999; 6(suppl 1):S58-S65.
56. Malone DE, MacEneaney PM. Applying ‘technology assessment’ and ‘evidence based medicine’ theory to interventional radiology. I. Suggestions for the phased evaluation of new procedures. Clin Radiol 2000; 55:929-937.
57. Dujardin B, Van den Ende J, Van Gompel A, Unger JP, Van der Stuyft P. Likelihood ratios, a real improvement for clinical decision making?. Eur J Epidemiol 1994; 10:29-36.[CrossRef][Medline]
58. Black WC, Welch HG. Advances in diagnostic imaging and overestimations of disease prevalence and the benefits of therapy. N Engl J Med 1993; 328:1237-1243.[Free Full Text]
59. Welch HG, Schwartz LM, Woloshin S. Are increasing 5-year survival rates evidence of success against cancer?. JAMA 2000; 283:2975-2978.[Abstract/Free Full Text]
60. Black WC, Welch HG. Screening for disease. AJR Am J Roentgenol 1997; 168:3-11.[Abstract/Free Full Text]
61. Prorok PC, Kramer BS, Gohagan JK. Screening theory and study design: the basics. In: Kramer BS, Gohagan JK, Prorok PC, eds. Cancer screening: theory and practice. New York, NY: Marcel Dekker, 1999; 29-53.
62. Hersh AL, Black WC, Tosteson AN. Estimating the population impact of an intervention: a decision-analytic approach. Stat Methods Med Res 1999; 8:311-330.[Abstract/Free Full Text]
63. Black WC. Should this patient be screened for cancer?. Eff Clin Pract 1999; 2:86-95.[Medline]
64. Lee JM. Screening and informed consent. N Engl J Med 1993; 328:438-440.[Free Full Text]
65. Wolf AM, Becker DM. Cancer screening and informed patient discussions: truth and consequences. Arch Intern Med 1996; 156:1069-1072.[Abstract]
66. Austoker J. Gaining informed consent for screening: is difficult—but many misconceptions need to be undone (editorial). BMJ 1999; 319:722-723.[Free Full Text]
67. MacEneaney PM, Malone DE. Applying "evidence-based medicine" theory to interventional radiology. II. A spreadsheet for swift assessment of procedural benefit and harm. Clin Radiol 2000; 55:938-945.
68. Dixon AK. Evidence-based diagnostic radiology. Lancet 1997; 350:509-512.[CrossRef][Medline]
69. Hersh W. "A world of knowledge at your fingertips": the promise, reality, and future directions of on-line information retrieval. Acad Med 1999; 74:240-243.[Medline]
70. Candy PC. Self-direction for lifelong learning: a comprehensive guide to theory and practice San Francisco, Calif: Jossey-Bass, 1991.
71. Abrahamson S, Baron J, Elstein AS, et al. Continuing medical education for life: eight principles. Acad Med 1999; 74:1288-1294.[Medline]
72. Slotnick HB. How doctors learn: physicians’ self-directed learning episodes. Acad Med 1999; 74:1106-1117.[Medline]
73. Davis D, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?. JAMA 1999; 282:867-874.[Abstract/Free Full Text]
74. Maudsley G. Do we all mean the same thing by "problem-based learning"? A review of the concepts and a formulation of the ground rules. Acad Med 1999; 74:178-185.[Medline]
75. Knowles MS. The modern practice of adult education: andragogy versus pedagogy New York, NY: New York Association, 1970.
76. Schon DA. Education the reflective practitioner: toward a new design for teaching and learning in the professions San Francisco, Calif: Jossey-Bass, 1990.
77. Guyatt G, Rennie D. Users’ guides to the medical literature: a manual for evidence-based clinical practice Chicago, Ill: American Medical Association, 2001.
78. Health Information Research Unit. Evidence-Based Health Informatics; Available at: hiru.mcmaster.ca/default.htm. Accessed August 18, 2000.
79. Hillman BJ. Outcomes research and cost-effectiveness analysis for diagnostic imaging (editorial). Radiology 1994; 193:307-310.[Medline]
80. ACRIN: American College of Radiology Imaging Network.; Available at: www.acrin.org. Accessed August 18, 2000.
81. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine New York, NY: Churchill Livingstone, 1997.
82. The Meaning of Diagnostic Test Results: A Spreadsheet for Swift Data Analysis.; Available at: ftp://radiography.com/pub/Rad-data99.xls. Accessed August 18, 2000.
83. Lusted LB. In the process of solution (editorial). N Engl J Med 1975; 292:255-256.[Medline]
84. Black WC. Anatomic extent of disease: a critical variable in reports of diagnostic accuracy (editorial). Radiology 2000; 217:319-320.[Free Full Text]
85. Gray JAM. Evidence-based healthcare: how to make health policy and management decisions Edinburgh, Scotland: Churchill Livingstone, 1997.
86. Auclair F. On the nature of evidence. Annals of the Royal College of Physicians and Surgeons of Canada 1999; 32:453-455.
87. Methodological issues in diagnostic clinical trials: health services and outcomes research in radiology—symposium proceedings. Acad Radiol 1999; 6(suppl 1):S1-S136.
88. Naylor CD. Grey zones of clinical practice: some limits of evidence based medicine. Lancet 1995; 345:840-842.[CrossRef][Medline]
89. Straus SE, McAlister FA. Evidence-based medicine: a commentary on common criticisms. CMAJ 2000; 163:837-841.[Abstract/Free Full Text]
90. Bruel JM, Malone D. Imaging of bowel obstruction (abstr). Eur Radiol 2000; 10(suppl 1):52.