Radiology
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

DOI: 10.1148/radiol.2422051679
This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Dodd, J. D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Dodd, J. D.
(Radiology 2007;242:342-354.)
© RSNA, 2007

Evidence-based Radiology Series

Evidence-based Practice in Radiology: Steps 3 and 4—Appraise and Apply Diagnostic Radiology Literature1

Jonathan D. Dodd, MD, MSc, MRCPI, FFR (RCSI)

1 From the Department of Radiology, Massachusetts General Hospital, Boston, Mass. Received October 12, 2005; revision requested November 14; revision received December 19; accepted January 19, 2006; final version accepted February 3. Address correspondence to the author, Department of Radiology, St Vincent's University Hospital, Elm Park, Dublin 4, Ireland (e-mail: j.dodd{at}st-vincents.ie).


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 EBP: THE STEPWISE PROCESS
 DISCUSSION
 APPENDIX
 References
 
Several paradigms for evidence-based practice (EBP) exist. One model proposes that specialist academic centers should primarily construct valid guidelines for various topics in medicine (top-down model). An alternative model integrates "the best research evidence with clinical expertise and patient values" (bottom-up model). Whereas the former model inherently implies a central specialized process, the latter implies that practitioners working in nonspecialist centers can learn and implement a standardized set of tools with which to ask a question, search and appraise the literature, and then apply best current evidence in a local setting. This article focuses on appraising the literature and applying retrieved results and is part of a series on EBP in radiology. This article describes a clinical scenario in which a new respirologist at a hospital requests indirect computed tomographic (CT) venography as part of a work-up of a patient with a high pretest probability for pulmonary embolism and a positive D-dimer test result. Many controversies surround the technique of indirect CT venography, and difficult topics such as this are ideally suited to the tools of EBP. This article will describe how to approach such a scenario.

© RSNA, 2007


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
EBP: THE STEPWISE PROCESS
 DISCUSSION
 APPENDIX
 References
 
The clinical scenario: You are a staff radiologist working in a busy tertiary referral academic hospital. A new respirologist has been recently appointed who has a well-established research interest in venous thromboembolism (VTE). During your weekly chest conference, he brings up the case of an elderly woman with a high pretest probability of pulmonary embolism (PE) and a positive finding at enzyme-linked immunosorbent assay D-dimer testing. Because the patient has moderately severe chronic obstructive airways disease, she is not a suitable candidate for a ventilation-perfusion study (1). The respirologist is requesting that computed tomographic (CT) pulmonary angiography be performed and that indirect CT venography be included as part of the protocol. He questions why this technique is not routinely performed in all patients suspected of having PE.

After the conference, the new respirologist writes to you in regard to the issue of indirect CT venography. In his letter, he emphasizes that "the prevalence of deep vein thrombosis (DVT) is 182 and that of PE is 201 per 100 000 observation-years (2). In-patient mortality from VTE has been estimated to be around 12% (3,4). VTE may be difficult to diagnose clinically, and clinical prediction models have been validated for both DVT and PE to help categorize patients as having low, intermediate, or high pretest probabilities for these conditions (5,6). Contemporary D-dimer testing has high sensitivity but low specificity (7). It has recently been demonstrated that use of indirect CT venography may increase the detection rate of VTE by as much as 20% (8). Given that approximately 90% of cases of PE arise from the pelvic or leg veins, it would seem prudent to include indirect CT venography as part of a CT pulmonary angiography protocol."

You write back to the respirologist, and in your letter you comment that "contrast material–enhanced venography is the classic reference standard for DVT (4). This examination is invasive, costly, causes considerable interobserver variation, and may induce thrombosis in a small number of cases (9,10). Ultrasonography (US) has a high sensitivity and specificity but also has limitations (10). It is poor at depicting the pelvic, iliac, and calf veins, is user-operator dependent, may result in false-negative findings in patients with duplicated venous anatomy, and has a lower accuracy in patients who are asymptomatic (11)." You finish your letter by suggesting that you undertake an evaluation of the literature on this topic, and you promise to get back to the respirologist with your results.

Your radiology resident is unfamiliar with indirect CT venography. You explain that it is a method whereby images from the lower limbs to the diaphragm are obtained, 2–4 minutes after CT pulmonary angiography, by using only the contrast material that is already in the circulation from the CT pulmonary angiographic examination (12). This allows visualization of the abdominal, pelvic, and leg veins. You suggest that your resident review the literature on the technique and arrange to discuss his findings at the next resident teaching rounds.

At rounds, the resident reports a frustratingly low yield of primary diagnostic articles. He initially searched with Google, which returned 777 Web sites (the first page of the search results contained two articles from the radiology literature). He then searched PubMed, which is a Web site developed by the National Center for Biotechnology Information and is designed to provide access to citations from the biomedical literature (13). By using the term indirect CT venography, he found five diagnostic articles in English (14–18), six review articles (19–24), three integrative epidemiology articles in English (8,25–27), one integrative epidemiology article in Italian (28), one article on the evaluation of cerebral venous thrombosis (29), one case report that described indirect azygous vein continuation syndrome in German (30), and one article on a cohort study of Budd-Chiari syndrome in Chinese (31). The five diagnostic articles that seemed relevant to indirect CT venography did not show homogeneous results. Sensitivities varied from 70% to 100% (15–18). Furthermore, CT scanning protocols were markedly different among the studies. The resident is not sure where to go from here.

You explain to the radiology resident that you have worked previously on difficult or "gray areas" of the radiology literature by using the principles of evidence-based medicine and applying them to your radiology practice. Such evidence-based practice (EBP) has its origins at McMaster University, Hamilton, Ontario, Canada, and the Oxford Centre for Evidence-Based Medicine, Oxford, England. Since its conception, EBP has been applied to many medical disciplines, including radiology. After your discussion with the respirologist, you decide that the tools of EBP would be ideally suited to this problem.

This article is part of a series on EBP in radiology. The first two articles provided an introduction to EBP (32,33), while the current article applies EBP tools to the evaluation of the diagnostic radiology literature by using indirect CT venography as an example.


   EBP: THE STEPWISE PROCESS
TOP
 ABSTRACT
 INTRODUCTION
 EBP: THE STEPWISE PROCESS
DISCUSSION
 APPENDIX
 References
 
There are five steps in applying the "evidence-based" approach (34): ask, search, appraise, apply, and evaluate. These steps were undertaken between July 2004 and November 2005 for the scenario presented in this article.

Step 1: Ask
Asking a focused question.—As discussed in the first article of this series (32), asking a focused clinical question involves using the four components of the PICO format: patient, investigation, comparison, and outcome of interest. These components are most useful if they are each used as a medical subject heading (MeSH). A search for suitable MeSH terms for any topic can be found by using the Preview/Index tab found on the PubMed home page (Fig 1).


Figure 1A
View larger version (29K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1a: MeSH terms in a search strategy with PubMed. (a) Clicking Preview/Index tab (arrow a) opens search box in all fields (arrow b). (b) Typing indirect CT venography in the search box and clicking the Index tab (arrow c) opens a list of possible MeSH terms (arrow d). Clicking most appropriate term will highlight it. Different search words can be typed in the search box. Clicking AND and OR tabs (arrow e) adds highlighted MeSH term to the Preview search. Individual searches can then be combined in PICO format. (c) Possible search strategy with MeSH terms. First search with indirect CT venography yields 13 articles (arrow f). Second search with tomography, x ray computed yields 161 371 articles (arrow g). Combining searches yields 10 articles (arrow h). Including only reports in English yields nine articles (arrow i). Such strategies result in more focused and relevant article retrieval.

 

Figure 1B
View larger version (39K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1b: MeSH terms in a search strategy with PubMed. (a) Clicking Preview/Index tab (arrow a) opens search box in all fields (arrow b). (b) Typing indirect CT venography in the search box and clicking the Index tab (arrow c) opens a list of possible MeSH terms (arrow d). Clicking most appropriate term will highlight it. Different search words can be typed in the search box. Clicking AND and OR tabs (arrow e) adds highlighted MeSH term to the Preview search. Individual searches can then be combined in PICO format. (c) Possible search strategy with MeSH terms. First search with indirect CT venography yields 13 articles (arrow f). Second search with tomography, x ray computed yields 161 371 articles (arrow g). Combining searches yields 10 articles (arrow h). Including only reports in English yields nine articles (arrow i). Such strategies result in more focused and relevant article retrieval.

 

Figure 1C
View larger version (32K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1c: MeSH terms in a search strategy with PubMed. (a) Clicking Preview/Index tab (arrow a) opens search box in all fields (arrow b). (b) Typing indirect CT venography in the search box and clicking the Index tab (arrow c) opens a list of possible MeSH terms (arrow d). Clicking most appropriate term will highlight it. Different search words can be typed in the search box. Clicking AND and OR tabs (arrow e) adds highlighted MeSH term to the Preview search. Individual searches can then be combined in PICO format. (c) Possible search strategy with MeSH terms. First search with indirect CT venography yields 13 articles (arrow f). Second search with tomography, x ray computed yields 161 371 articles (arrow g). Combining searches yields 10 articles (arrow h). Including only reports in English yields nine articles (arrow i). Such strategies result in more focused and relevant article retrieval.

 
Result of asking a focused question on indirect CT venography.—In text format, the question would read as follows: In patients suspected of having VTE, how does indirect CT venography compare with a reference standard in the diagnosis of DVT?

Step 2: Search
EBP search strategies (32) were used.

Levels of evidence.—There is a hierarchy of evidence in the literature (35). At its most basic level, the medical literature can be divided into primary and secondary evidence. At the top of the "evidence pyramid" (35) are evidence-based guidelines that summarize important and relevant topics in clinical medicine. One such system is Clinical Evidence from the British Medical Journal Publishing Group (36). The next level of evidence is made up of evidence-based journals, such as the American College of Physicians Journal Club (37). The next level includes evidence-based reviews, guidelines, and databases—for example, the Cochrane Collaboration (38). All of these levels together make up the secondary literature. For the clinical scenario described above, each of these sources was searched for articles by using the term indirect CT venography.

The primary literature consists of articles on original studies and is the lowest level on the evidence pyramid. For this level of evidence, the PubMed Web site was searched by using the PICO format, which allows one to link concepts in a search strategy. The use of Boolean operator terms in the search bar on the PubMed Web site allows one to link similar concept terms by using the operator OR and different concept terms by using the operator AND (Fig 1). For the current clinical scenario, the primary literature was searched by using the MeSH terms listed in Figure 2. An example of one possible search strategy for the current scenario is given in Figure 1c.


Figure 2
View larger version (6K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2: Possible search strategies with the PICO format. The search can be customized to suit different requirements. If time is limited, searching MeSH terms across the row will result in a small retrieval of the most important articles but may miss some relevant ones. If more time is available or a search is required to encompass all important articles on a topic, the search can be widened by inserting more column MeSH terms. This will result in a larger retrieval of important articles but may include some less relevant ones.

 
Results from searching the literature.—A search of evidence-based Web sites revealed that under the topic of VTE there were separate recommendations for PE and invasive venography but none for indirect CT venography. For the search of evidence-based journals, the author did not subscribe to American College of Physicians Journal Club, and there was no free symposium on indirect CT venography. A search of the Cochrane Library, Guidelines Finder, and the Scottish Intercollegiate Guidelines Network revealed no reviews on indirect CT venography. A search performed by using the search engine SUMSearch yielded no systematic reviews.

During the PubMed search of the primary literature, two different groups of articles became evident. One group consisted of a large number of articles of integrative epidemiology studies performed to assess aspects of indirect CT venography other than diagnostic evaluation. These articles were not the primary focus of the current clinical scenario and were not appraised in detail.

The second group consisted of articles on primary diagnostic studies, and this group of reports was appraised in detail (12,14–17,39–48). One article on the evaluation of indirect CT venography in the intensive care setting was excluded, since (a) patients in such a setting have many different clinical aspects compared with patients in a non–intensive care setting, (b) clinical rather than technologic outcomes were used as the reference standard, and (c) such a setting is unrelated to the clinical scenario being used in this report (18). The data from the excluded study did support the use of indirect CT venography as an alternative to US in intensive care patients.

The retrieved articles are listed in Table 1. The Related Articles tab on the PubMed Web site is another possible search option, but the author did not find this feature to be very specific. For example, clicking on the Related Articles tab for each retrieved article yielded only two retrievals (ie, entire sets of related articles for a given topic) with fewer than 30 articles, many of which were unrelated to indirect CT venography.


View this table:
[in this window]
[in a new window]

  		Table 1. Retrieved Articles for Clinical Scenario and Number of Articles Retrievable with the PubMed Related Articles Tab

 
Readers interested in more detailed discussion on searching the literature are referred to the first article in this series (32).

Step 3: Appraise
Applying levels of evidence to articles.—How does a busy radiologist sift through large numbers of articles to find those that have the least amount of bias—and therefore provide results that are closest to the truth? Among many options, one of the most useful is to assign a level of evidence to each article. In practice settings where local EBP groups exist, assigning levels of evidence can be a useful teaching exercise for students, residents, and fellows and can save the senior radiologist valuable time. Practical aspects of teaching EBP will be discussed in a later article in this series.

The author's local evidence-based radiology group consists of residents and fellows who meet once a week to learn about and apply the principles of EBP to their local radiology practice. Meetings are voluntary and are led by a consultant (attending) radiologist who provides teaching about EBP and resources for solving difficult radiology problems. For junior members, this results in substantial contributions to local practice and stimulates abstract publication at international academic meetings and article publication in the international peer-reviewed literature (39–41).

The Oxford Centre for Evidence-Based Medicine has developed a table of levels of evidence (Table 2 shows an abbreviated form of the levels of evidence; the full table is freely accessible online) (42). With this table, one can quickly assign a level of evidence to each article retrieved from PubMed. This is useful because articles that are assigned a very low level of evidence do not require further analysis unless it is the best evidence available. In this way, only the highest level of articles need to be evaluated, which can markedly reduce the reading load. There are levels of evidence not just for articles on diagnostic studies but also for articles on therapy, prevention, harm, etiology, prognosis, differential diagnosis, and economic and decision analysis studies. Discussion of these is beyond the focus of the current report, but interested readers are referred to the Web site of the Oxford Centre for Evidence-Based Medicine for further information (42).


View this table:
[in this window]
[in a new window]

  		Table 2. Levels of Evidence and Grades of Recommendations for Diagnostic Studies from the Oxford Centre for Evidence-Based Medicine

 
Results from assigning a level of evidence.—The results of assigning levels of evidence can be seen in Table 3. There were no level 1a, 2a, or 3a systematic reviews for the evaluation of indirect CT venography. There were two level 1b articles, two level 2b articles, two level 3b articles, and eight level 4 articles. The level 2b articles did not state that consecutive patients were used. The level 3b articles did not have the stated reference standard applied to all patients. Several methodological flaws were present in the studies in the level 4 reports, and the commonest flaw was the nonblinded performing of tests (or not stated in the articles). The level 1b articles were the highest level reports and were appraised in the most detail.


View this table:
[in this window]
[in a new window]

  		Table 3. Levels of Evidence for the Retrieved Articles on Indirect CT Venography

 
Evaluating validity from the Materials and Methods section.—When an article from the diagnostic literature is appraised, two sections are evaluated. The Materials and Methods section of the article is assessed for the "validity" of the study, and the Results section is evaluated for the statistical "strength" of the study.

Several standard questions are asked when one is appraising a diagnostic study for validity (34): (a) Was there an independent, blinded comparison with a reference standard of diagnosis? (b) Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in whom it would be used in practice)? (c) Was the reference standard applied regardless of the diagnostic test result? (d) Was the test (or a cluster of tests) validated in a second, independent group of patients?

Each time the answer to a question is "no," a potential source of methodological bias is identified. The Centre for Evidence-Based Medicine has stated that the ideal study performed to evaluate a diagnostic test should be an independent blinded comparison of an appropriate spectrum of consecutive patients, all of whom have undergone both the diagnostic test and the reference standard (42).

Results from appraising validity in the retrieved articles.—The results from appraising the validity in the retrieved articles in this report can be seen in Table 4. Both of the retrieved level 1b studies had an independent blinded comparison with a reference standard in an appropriate spectrum of patients (17,43). The authors did not validate the test in a second completely independent group of patients in either of these studies.


View this table:
[in this window]
[in a new window]

  		Table 4. Validity of the Highest Ranked Studies Derived from the Materials and Methods Section of Each Article

 
Additional points for a radiologist to consider.—It is suggested that the Materials and Methods section of radiology articles, in addition to being used to answer epidemiologic questions, should be appraised from the radiologist's perspective with five further questions. These questions are as follows (41): (a) Has the imaging method been described in sufficient detail for it to be reproduced in one's own department? (b) Has the imaging test been evaluated and the reference test been performed to the same standard of excellence? (c) Have "generations" of technology development within the same modality (eg, conventional vs helical single–detector row vs multi–detector row CT) been adequately considered in the study design and discussion? (d) Has radiation exposure been considered? (The concept of justification and optimization has assumed prime importance in radiation protection to patients.) (e) Were magnetic resonance and/or CT images reviewed on a monitor or as film (hard-copy) images? For example, with indirect CT venography, small clots may be difficult to see on film images but may become visible on digital images by means of careful manipulation of window width and level settings (44).

Results from appraising validity from a radiologist's perspective.—Many radiologic aspects of indirect CT venography were not included in the two level 1b articles (17,43). Details on the kilovolt peak, milliampere-second, window width, and window center were omitted from one or the other of the articles (Table 5), which makes these protocols difficult to reproduce in other departments (Table 6). Technology generations were not discussed in one of the level 1b articles (43). Radiation exposure was not considered in either article. In one of the articles (43), it was not stated whether film or digital images were used in evaluation.


View this table:
[in this window]
[in a new window]

  		Table 5. Indirect CT Venography Protocols from the Highest Ranked Studies

 

View this table:
[in this window]
[in a new window]

  		Table 6. Validity of the Highest Ranked Studies: Additional Points for Radiologists to Consider

 
Because of the "radiologic" limitations in the level 1b articles, the two retrieved level 2b articles (16,45) were also appraised in more detail. In one of the level 2b articles (16), the kilovolt peak and milliampere-second values were stated for the indirect CT venography examination. In both studies, the imaging test and the reference standard were performed to the same level. In one of the level 2b articles (16), the authors discussed technology generations and evaluated radiation exposure. The same authors stated that digital images were read.

Appraising strength from the Results section.—Assessment of the strength of a radiology study can be found in the Results section of an article. The important statistical parameters include the prevalence of DVT, sensitivity and specificity (with 95% confidence intervals), predictive values, and likelihood ratios (LRs). Further reference to the derivations of these calculations can be found in the Appendix. Any articles that did not have usable raw data were discarded. The data (true-positive, true-negative, false-positive, and false-negative findings) from all four appraised articles were pooled in order to calculate an unweighted summary estimate of sensitivity, specificity, predictive values, and LRs for indirect CT venography.

Findings from assessment of the Results section.—Three articles (16,17,43) demonstrated high sensitivity and specificity for indirect CT venography in the detection of DVT (Table 7). One article (45) demonstrated a lower sensitivity (sensitivity of 0.71). Confidence intervals in three articles were satisfactorily narrow (16,17,43). One article showed wide confidence intervals for sensitivity (95% confidence interval: 0.5, 0.95) (45). The positive predictive value was very high in two articles (17,43) (VTE prevalence, 100% and 27%, respectively), moderately high in one article (16) (VTE prevalence, 49%), and low in one article (45) (VTE prevalence, 10%). The negative predictive value was high in all four articles. For three of the retrieved articles the positive LR was greater than 10 and the negative LR was less than 0.1; theses ratios indicated a strong likelihood of disease being present if the test result was positive and of the disease being absent if the test result was negative (16,17,43). For one article (45), the positive LR was 9.6 and the negative LR was 0.3, and these values indicated a moderate likelihood of disease being present if the test result was positive and of the disease being absent if the result was negative. A hip prosthesis in one patient and a below-knee amputation in another patient led to misinterpretation of the indirect CT venograms and contributed substantially to the poorer statistical performance in one of the level 2b articles (45).


View this table:
[in this window]
[in a new window]

  		Table 7. Strength of the Highest Ranked Studies Derived from the Results Section of Each Article

 
Step 4: Apply
Combining LRs and pretest probabilities: graphs of conditional probability.—One of the most useful things about LRs is that they may be combined with the pretest odds for a disease to give the posttest probability (46,47). Remember that all that is required to calculate LRs is the sensitivity and specificity of a test. The entire spectrum of pretest odds for a disease can then be multiplied by the LR of a particular test to give a corresponding spectrum of posttest probabilities. This can be graphically represented and is called a graph of conditional probability (GCP). The calculations of these graphs can be performed from spreadsheets (sources for these and an explanation of how GCPs were derived for the current article can be found in the Appendix). GCPs help to answer the questions posed in the current clinical scenario. It is the posttest probability that the clinician really wants to know: "If I order indirect CT venography to be performed in this patient, given the imaging result, what now is the posttest probability that my patient has DVT?" An integral part of using these GCPs is estimating the pretest probability of disease. For populations, this is the disease prevalence, and for individual patients, responsibility for the pretest probability lies both with the referring clinician who should provide it and with the radiologist who should ask for it. For VTE, clinical decision rules have been derived for particular symptoms, signs, and risk factors (5,6).

Results from using GCPs: current clinical scenario.—In the current clinical scenario, an elderly woman has a high pretest probability of PE and a positive enzyme-linked immunosorbent assay D-dimer test result. For patients with a high pretest clinical probability of PE, the prevalence is 78% (6). If this prevalence is applied as the pretest probability to the GCP for D-dimer testing and the result is positive, the posttest probability is 85%, which warrants further investigation (Fig 3a). This posttest probability for the D-dimer test can be used as the pretest probability for another investigation, provided that this investigation is not measuring the same disease property.


Figure 3A
View larger version (27K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3a: Use of GCPs to achieve clinical resolution (solid lines = positive result, dashed lines = negative result). Posttest probability for a positive result is derived by drawing a vertical line up to the solid curved line and then across to the y-axis. Posttest probability for a negative result is derived by drawing a vertical line up to the dotted curved line and then across to the y-axis. (a) GCP for D-dimer test. For a patient with a high pretest probability of PE, the prevalence is 78% (solid arrow). Posttest probability for a positive D-dimer result is 85% (open arrow), which warrants further investigation. (b) This posttest probability is then applied as pretest probability to the GCP for CT pulmonary angiography (solid arrow). If the result is positive, posttest probability is 99% (open arrow) and PE is confirmed. If the result is negative, posttest probability is 30% (curved arrow), which is not low enough to exclude disease (further investigation warranted). (c) This posttest probability is then applied as pretest probability to the GCP for indirect CT venography (solid arrow). If the result is positive, posttest probability of DVT is greater than 72% (open arrow) and diagnosis is confirmed. If the result is negative, posttest probability of DVT is less than 5% (curved arrow) and the diagnosis is excluded.

 

Figure 3B
View larger version (28K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3b: Use of GCPs to achieve clinical resolution (solid lines = positive result, dashed lines = negative result). Posttest probability for a positive result is derived by drawing a vertical line up to the solid curved line and then across to the y-axis. Posttest probability for a negative result is derived by drawing a vertical line up to the dotted curved line and then across to the y-axis. (a) GCP for D-dimer test. For a patient with a high pretest probability of PE, the prevalence is 78% (solid arrow). Posttest probability for a positive D-dimer result is 85% (open arrow), which warrants further investigation. (b) This posttest probability is then applied as pretest probability to the GCP for CT pulmonary angiography (solid arrow). If the result is positive, posttest probability is 99% (open arrow) and PE is confirmed. If the result is negative, posttest probability is 30% (curved arrow), which is not low enough to exclude disease (further investigation warranted). (c) This posttest probability is then applied as pretest probability to the GCP for indirect CT venography (solid arrow). If the result is positive, posttest probability of DVT is greater than 72% (open arrow) and diagnosis is confirmed. If the result is negative, posttest probability of DVT is less than 5% (curved arrow) and the diagnosis is excluded.

 

Figure 3C
View larger version (27K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3c: Use of GCPs to achieve clinical resolution (solid lines = positive result, dashed lines = negative result). Posttest probability for a positive result is derived by drawing a vertical line up to the solid curved line and then across to the y-axis. Posttest probability for a negative result is derived by drawing a vertical line up to the dotted curved line and then across to the y-axis. (a) GCP for D-dimer test. For a patient with a high pretest probability of PE, the prevalence is 78% (solid arrow). Posttest probability for a positive D-dimer result is 85% (open arrow), which warrants further investigation. (b) This posttest probability is then applied as pretest probability to the GCP for CT pulmonary angiography (solid arrow). If the result is positive, posttest probability is 99% (open arrow) and PE is confirmed. If the result is negative, posttest probability is 30% (curved arrow), which is not low enough to exclude disease (further investigation warranted). (c) This posttest probability is then applied as pretest probability to the GCP for indirect CT venography (solid arrow). If the result is positive, posttest probability of DVT is greater than 72% (open arrow) and diagnosis is confirmed. If the result is negative, posttest probability of DVT is less than 5% (curved arrow) and the diagnosis is excluded.

 
The posttest probability from the positive D-dimer result (85%) is applied as the pretest probability to the GCP for CT pulmonary angiography (Fig 3b) (48,57). If the result is positive, the posttest probability is 99% and the diagnosis is confirmed. If the result is negative, the posttest probability is 30%, which is not low enough to exclude disease. This posttest probability for CT pulmonary angiography can then be used as the pretest probability for the GCP for indirect CT venography.

The posttest probability from the negative result at CT pulmonary angiography (30%) is applied as the pretest probability to the GCP for indirect CT venography (Fig 3c). If the test result is positive, the posttest probability of DVT is greater than 72% and the diagnosis is confirmed. If the test result is negative, the posttest probability of DVT is less than 5% and the diagnosis is excluded.

Clinical Resolution
Now return to the clinical scenario described in the beginning of this article. You present your "evidence-based" approach to indirect CT venography to your radiology colleagues, who agree with your findings. You write to the respirologist with your results: "Current best evidence (from primary diagnostic studies) shows that indirect CT venography has a high sensitivity and specificity with satisfactorily narrow confidence intervals for the detection of DVT. In certain patient subgroups (those with hip prosthesis and below-knee amputations), there is a substantially lower sensitivity and specificity, and these patients may be more optimally evaluated with US. Findings of integrative epidemiologic studies suggest that using helical multi–detector row CT with a scan range that includes the midabdomen and pelvis may increase the detection of acute VTE by up to 20%. We recommend its routine implementation in patients with intermediate or high pretest probability. Findings of two studies have confirmed an increased radiation dose to patients specifically from indirect CT venography. In patients with low probability, particularly those younger than 40 years, the need for indirect CT venography should be reviewed on a case-by-case basis and an alternative investigation, such as bilateral leg US, may be undertaken to rule out DVT."

Step 5: Evaluate
The final step incumbent for EBP practitioners is to evaluate their results in clinical practice. Outcomes obtained in specialist centers can then be evaluated locally. The author's local radiology group has recently implemented indirect CT venography into departmental protocols (58).

The author's results may differ from those in previous articles for many potential reasons. Such translation from an efficacy level to an effectiveness level has been recognized by others (59). We found that a learning curve exists for indirect CT venography in clinical practice, and this is similar to the findings of other groups (60). Radiologists should not expect indirect CT venography to generate images with a peak venous enhancement similar to peak pulmonary arterial enhancement. Two excellent pictorial reviews (44,61) that highlight the potential pitfalls in image interpretation may be useful during the initial introduction of this technique to local practice. These reviews also describe several technical parameters not included in earlier work.


   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 EBP: THE STEPWISE PROCESS
 DISCUSSION
APPENDIX
 References
 
Indirect CT venography is a relatively new technique that allows direct visualization of thrombi in the deep venous system of the abdomen, pelvis, and lower limbs. Because the technique is contemporary, many traditional sources, such as radiology textbooks, have limited information on diagnostic performance. Colleagues may be unfamiliar with its methods. It is a rapidly evolving technique that continues to undergo refinements. Expert opinion is divided on many aspects of indirect CT venography, and such difficult topics are ideally suited for the tools of EBP. These tools can help radiologists sift through what is often a literary quagmire and achieve resolution for such gray areas of the literature. Traditional use of purely subjective opinion is receding, and the ability to search for and appraise articles by using systematic methods is assuming greater importance. This is encompassed in the EBP paradigm, which is defined as "the integration of best research evidence with clinical expertise and patient values" (34). The shift in medical practice is particularly relevant to radiology, where technologies characteristically evolve rapidly. Now, more than ever, radiologists must maintain their knowledge base. The EBP analysis of the literature on indirect CT venography performed here revealed some important issues.

Indirect CT Venography Protocol
Scan delay.—Reported scan delays vary widely in the literature—anywhere between 2 and 4 minutes (15,49). Authors of one level 1b study used a 180-second scan delay and reported no inconclusive scans (17). Authors of the other level 1b study used a 210-second delay, with similar results (49). In one level 2b study (16), authors used a 180-second scan delay and reported two scans that were poor and one that was unreadable. In the other level 2b study (45), there were seven scans with poor venous contrast agent opacification. The two causes of poor opacification in this study were streak artifact from hip fixation hardware and a below-knee amputation that caused asymmetrical opacification of leg veins.

The use of a shorter scan delay appears to result in higher numbers of suboptimal scans. In a larger series of 541 patients suspected of having PE, a scan delay of 120 seconds resulted in 23% of indirect CT venograms being rated as "fair to poor" (15). Thus, results the analysis in this report would support a longer scan delay of 180 seconds or more. In a recent study (62) in which authors used calf and thigh stockings to increase venous flow, findings indicated a 30% increase in opacification of the deep venous system. This addition to conventional protocols should improve image quality further. The use of multi–detector row CT is likely to influence the scan delay in future studies. More rapid rates of contrast agent injection are being used with current multi–detector row CT protocols, which may lead to shorter delays before opacification of the pelvic and leg veins (63).

Section interval.—In one level 1b study (43), a section interval of 50 mm was used, and no DVT was missed with this protocol. In a follow-up study of 650 patients (49), 308 of whom underwent comparison at US, there were two false-negative findings with indirect CT venography, which were related to short-segment thrombus in the superficial and common femoral veins. More recently Cham et al (8) demonstrated in a study of 1590 patients that among the 148 patients with DVT, 24% had thrombi that were less than 50 mm in length. In two of our retrieved multi–detector row CT studies in which section intervals of 5 and 8 mm were used, sensitivities were 100%. Thus, results of our analysis would support a section interval narrower than 50 mm. This would increase the number of images acquired and the time required to read them. In this regard, the use of multi–detector row CT is likely to have a strong effect. In one of the level 1b studies (43), in which single–detector row CT was performed with a section width of 5–10 mm and a section interval of 50 mm, 5–7 minutes was required to obtain the indirect CT venogram. In the other level 1b study (17), in which multi–detector row CT was performed with a section width of 5 mm and a section interval of 5 mm, 80–90 seconds was required to obtain the indirect CT venogram. In a more recent study (27) in which six different multi–detector row CT protocols were compared, findings suggested that the optimum section interval was 6 mm. Although the number of images that need to be acquired is increased, this must be countered against an improved detection rate of VTE, which may justify the added time required to read images.

Scan range.—The four studies appraised in the retrieval in this report had relatively homogeneous scan ranges, but other studies have used shorter ranges—for example, from the midcalves to the acetabulum (25). In both level 1b studies (17,43), six of 19 patients with DVT had a thrombus in the iliac veins or inferior vena cava. In a larger cohort of 650 patients suspected of having VTE, 31 patients had DVT without evidence of PE at indirect CT venography, and 10 of the thrombi were located in the inferior vena cava (49). Thus, results of this analysis would support a scan range that includes the midabdomen and pelvis. Longer scan ranges have the added benefit of enabling detection of additional nonthrombotic findings, albeit in small numbers. In one study (56) on the use of indirect CT venography, ascites, a pelvic mass, a hepatic and retrocaval mass, hepatic metastases, and a renal mass were detected.

Radiation dose.—It is striking that of the 14 primary diagnostic articles retrieved for this report, in only one was radiation dose specifically examined (16). In that article, collimations of 2.5 and 5.0 mm with a pitch of 1.25 mm gave effective total doses of 8.26 and 7.25 mSv, respectively. Rademaker et al (64) measured effective and gonadal dose in six patients with thermoluminescence dosimeters. With use of a collimation of 8 mm, the effective dose was 2.3–2.7 mSv. The ovarian dose from combined CT pulmonary angiography and indirect CT venography was 4.7 mSv, and the testicular dose was 6.7 mSv. Evolving multi–detector row CT dose reduction techniques—notably, automatic tube current modulation—result in substantial reductions in radiation dose (65). Wider collimation and lower milliampere-second values than those used by Begemann et al (16) should reduce doses further.

Whether the radiation risk inherent to indirect CT venography outweighs the benefit of disease detection must be assessed for each individual patient. Those patients with an intermediate or high pretest probability have an actual prevalence of 28% or 78% for PE, respectively (5). Patients with a low pretest probability have an actual prevalence of 3% for PE, and alternative methods of evaluation such as bilateral leg US may be more prudent in this group, particularly in those patients younger than 40 years (66). In this regard, Stein et al (67) reviewed the National Hospital Discharge Survey for the entire United States and found that the rates of diagnosis of DVT and PE and the use of diagnostic tests over 21 years were markedly higher in elderly patients compared with younger patients. Although the rate of diagnosed DVT in elderly patients increased strikingly over the past decade, that of PE remained relatively constant. It is likely in future years that the majority of indirect CT venography examinations will be performed for elderly patients.

Practicing EBP in Radiology
Points for radiologists to consider.—Medical epidemiologists designed the tools of EBP for medical articles. Technology in radiology has become progressively more complex, and certain aspects are not easily comprehensible to nonradiologists. Some additional parameters to the medical EBP checklist have been added here in an attempt to include an appraisal of "radiologic" aspects in study methods (41). This proved to be an important section in the current article. Neither level 1b article provided all CT parameters required to repeat the technique in another department. Because of this, the retrieved level 2b articles were also appraised. This additional appraisal is acceptable if information is unavailable from higher level articles and is obtainable from lower level articles if it represents the best current evidence available. In the current appraisal, the two level 2b articles provided important aspects of multi–detector row CT, such as kilovolt peak and milliampere-second values, that were not included in the level 1b articles.

Lack of standardization in study methods.—The tables of evidence from the Oxford Centre for Evidence-Based Medicine were used because they are freely accessible, are conceptually easy to comprehend and apply, and are internationally recognized as robust. It must be stressed that alternative methods for appraisal of articles exist.

Several of the articles in the initial retrieval were well designed in many aspects but omitted one or two important epidemiologic requirements from the text. In one instance (51), the study used technologists who were blinded to the results of indirect CT venography, but this fact was not stated in the article (V. W. Au, written communication). Lack of standardization caused a lower level of evidence to be assigned to several studies, many of which are immense research undertakings and achievements and may not contain any "real-life" methodological flaws. Such lack of standardization creates a barrier to the application of EBP to radiology. The Standards for Reporting of Diagnostic Accuracy Initiative attempts to implement consistency in study design by providing a 25-item checklist with which researchers can construct epidemiologically sound diagnostic research (68).

The need for such explicitness in study design was recently highlighted by Schmidt et al (69), who evaluated Medline for English-language articles published in 2000 in journals with an impact factor of greater than 4. Only 41% of articles reported on more than 50% of the Standards for Reporting of Diagnostic Accuracy items, and no article reported on more than 80% of the items. The technology assessment and EBP paradigms will be considered in detail later in this series. The importance of explicitness is highlighted by the American College of Radiology Task Force on Appropriateness Criteria, which regularly publishes nationally accepted scientifically derived guidelines for various disease topics (70). Such practice guidelines are generated through a rigorous evaluation of the literature by an expert panel (interested readers are referred to the American College of Radiology for further reading [70]). The pros and cons of "expert" panel evidence will be considered in a later article in this series.


   APPENDIX
TOP
 ABSTRACT
 INTRODUCTION
 EBP: THE STEPWISE PROCESS
 DISCUSSION
 APPENDIX
References
 
Sensitivity is calculated as TP/(TP + FN) and specificity is calculated as TN/(TN + FP), where FN, FP, TN, and TP are the numbers of false-negative, false-positive, true-negative, and true-positive findings, respectively. Results should be presented with appropriate indicators of measurement error or uncertainty such as confidence intervals. A 95% confidence interval is the range of values within which the true result will lie 95% of the time. Interested readers can refer to additional articles for further reading (47,71,72).

LR is the ratio of two probabilities: the probability of a test result in patients with disease divided by the probability of the same result in disease-free patients. LRs are calculated from sensitivity and specificity (47): the LR of a positive result is determined as sensitivity/(1 – specificity) and the LR of a negative result is determined as (1 – sensitivity)/specificity. LRs summarize the information in both sensitivity and specificity and convey the discriminative power of a test numerically (73). LRs lie in the range of 0 to infinity. A LR of 0 excludes disease—the result is never found in the patients with disease. A LR of infinity confirms disease—the result is never found in disease-free patients. If the LR equals 1, it implies that the test result is found equally in both patients with disease and disease-free patients; therefore, the test has no discriminating power for this indication. In practice, LRs of greater than 10.0 or less than 0.1 are considered strongly positive for disease and strongly negative for disease, whereas LRs in the range from 2.0 to 0.5 are weakly positive to weakly negative, respectively. The closer the LR is to 1.0, the lower the effect the result has on disease probability. Interested readers should refer to articles by MacEneaney and Malone (47) and by Radack et al (74).

For pretest probabilities, clinicians do not use mathematic formulas but develop a general sense of "probability" classified in broad terms as low, intermediate, or high pretest probability of having a given diagnosis. The spectrum of pretest probabilities for a given disease ranges from 0% pretest probability (the clinician is sure the patient does not have the diagnosis) to 100% pretest probability (the clinician is sure the patient has the diagnosis). Alternatively the clinician may be entirely uncertain of the diagnosis (50% pretest probability). Use of GCPs can help interpret such clinical pretest probabilities by multiplying the pretest odds by the LR to give the posttest probability. In the current report, the GCPs for D-dimer and CT pulmonary angiography were calculated from sensitivities and specificities derived from methodologically sound systematic reviews in the literature (48,57). A spreadsheet that is widely available on the Internet calculates all of the above statistical parameters automatically, is easily used, and is downloadable free in either laptop or personal digital assistant format (75). Once statistical parameters have been calculated, they can be summarized and stored by using critically appraised topics. These can also be freely downloaded from the Internet (76).


   ACKNOWLEDGMENTS
 
The author acknowledges the extensive advice and assistance of Dermot Malone, series editor, in the writing of the manuscript.


   FOOTNOTES
 

Abbreviations: DVT = deep vein thrombosis • EBP = evidence-based practice • GCP = graph of conditional probability • LR = likelihood ratio • MeSH = medical subject heading • PE = pulmonary embolism • VTE = venous thromboembolism

Author stated no financial relationship to disclose.


   References
TOP
 ABSTRACT
 INTRODUCTION
 EBP: THE STEPWISE PROCESS
 DISCUSSION
 APPENDIX
 References
 

  1. Calvo-Romero JM, Lima-Rodriguez EM, Bureo-Dacal P, Perez-Miranda M. Predictors of an intermediate ventilation/perfusion lung scan in patients with suspected acute pulmonary embolism. Eur J Emerg Med 2005;12:129–131.[CrossRef][Medline]
  2. Hansson PO, Welin L, Tibblin G, Eriksson H. Deep vein thrombosis and pulmonary embolism in the general population: the study of men born in 1913. Arch Intern Med 1997;157:1665–1670.[Abstract]
  3. Anderson FA Jr, Wheeler HB, Goldberg RJ, et al. A population-based perspective of the hospital incidence and case-fatality rates of deep vein thrombosis and pulmonary embolism: the Worcester DVT Study. Arch Intern Med 1991;151:933–938.[Abstract]
  4. Hyers TM. Venous thromboembolism. Am J Respir Crit Care Med 1999;159:1–14.[Free Full Text]
  5. Wells PS. Use of a clinical model for safe management of patients with suspected pulmonary embolism. Ann Intern Med 1998;129:997–1005.[Abstract/Free Full Text]
  6. Wells PS, Anderson DR, Bormanis J, et al. Value of assessment of pretest probability of deep-vein thrombosis in clinical management. Lancet 1997;350:1795–1798.[CrossRef][Medline]
  7. Stein PD, Hull RD, Patel KC, et al. D-Dimer for the exclusion of acute venous thrombosis and pulmonary embolism: a systematic review. Ann Intern Med 2004;140:589–602.[Abstract/Free Full Text]
  8. Cham MD, Yankelevitz DF, Henschke CI. Thromboembolic disease detection at indirect CT venography versus CT pulmonary angiography. Radiology 2005;234:591–594.[Abstract/Free Full Text]
  9. Kalodiki E, Nicolaides AN, Al-Kutoubi A, Cunningham DA, Mandalia S. How "gold" is the standard? interobservers' variation on venograms. Int Angiol 1998;17:83–88.[Medline]
  10. Tapson VF, Carroll BA, Davidson BL, et al; for the American Thoracic Society. The diagnostic approach to acute venous thromboembolism: clinical practice guideline. Am J Respir Crit Care Med 1999;160:1043–1066.
  11. Fraser JD, Anderson DR. Deep venous thrombosis: recent advances and optimal investigation with US. Radiology 1999;211:9–24.[Free Full Text]
  12. Loud PA, Grossman ZD, Klippenstein DL, Ray CE. Combined CT venography and pulmonary angiography: a new diagnostic technique for suspected thromboembolic disease. AJR Am J Roentgenol 1998;170:951–954.[Free Full Text]
  13. NCBI PubMed Web site. U.S. National Library of Medicine. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi. Accessed October 10, 2005.
  14. Yoshida S, Akiba H, Tamakawa M, Yama N, Takeda M, Hareyama M. Spiral CT venography of the lower extremities by injection via an arm vein in patients with leg swelling. Br J Radiol 2001;74:1013–1016.[Abstract/Free Full Text]
  15. Cham MD, Yankelevitz DF, Shaham D, et al; for the Pulmonary Angiography–Indirect CT Venography Cooperative Group. Deep venous thrombosis: detection by using indirect CT venography. Radiology 2000;216:744–751.
  16. Begemann PG, Bonacker M, Kemper J, et al. Evaluation of the deep venous system in patients with suspected pulmonary embolism with multi-detector CT: a prospective study in comparison to Doppler sonography. J Comput Assist Tomogr 2003;27:399–409.[CrossRef][Medline]
  17. Lim KE, Hsu WC, Hsu YY, Chu PH, Ng CJ. Deep venous thrombosis: comparison of indirect multidetector CT venography and sonography of lower extremities in 26 patients. Clin Imaging 2004;28:439–444.[CrossRef][Medline]
  18. Taffoni MJ, Ravenel JG, Ackerman SJ. Prospective comparison of indirect CT venography versus venous sonography in ICU patients. AJR Am J Roentgenol 2005;185:457–462.[Abstract/Free Full Text]
  19. Ciccotosto C, Goodman LR, Washington L, Quiroz FA. Indirect CT venography following CT pulmonary angiography: spectrum of CT findings. J Thorac Imaging 2002;17:18–27.[CrossRef][Medline]
  20. Gulsun M, Goodman LR, Washington L. Venous thromboembolic disease: where does multidetector computed tomography fit? Cardiol Clin 2003;21:631–638.[CrossRef][Medline]
  21. Reid JH. Multislice CT pulmonary angiography and CT venography. Br J Radiol 2004;77(suppl 1):S39–S45.[Abstract/Free Full Text]
  22. Wildberger JE, Mahnken AH, Das M, Kuttner A, Lell M, Gunther RW. CT imaging in acute pulmonary embolism: diagnostic strategies. Eur Radiol 2005;15:919–929.[CrossRef][Medline]
  23. Patel S, Kazerooni EA. Helical CT for the evaluation of acute pulmonary embolism. AJR Am J Roentgenol 2005;185:135–149.[Abstract/Free Full Text]
  24. Czekajska-Chehab E, Drop A, Terlecka B, Trzeciak J, Trojanowska A, Odoj M. Indirect CT venography of the abdominal cavity and lower limbs in patients with the suspicion of pulmonary embolism: indications, technique, diagnostic possibilities. Ann Univ Mariae Curie Sklodowska (Med) 2004;59:508–518.[Medline]
  25. Richman PB, Wood J, Kasper DM, et al. Contribution of indirect computed tomography venography to computed tomography angiography of the chest for the diagnosis of thromboembolic disease in two United States emergency departments. J Thromb Haemost 2003;1:652–657.[CrossRef][Medline]
  26. Chen SQ, Zhang TT, Kang Z, Zhang JS, Lin YY. Prediction of thrombolytic therapy for acute venous thromboembolic disease by CT pulmonary angiography and indirect CT venography. Chin Med J (Engl) 2004;117:1095–1097.[Medline]
  27. Das M, Muhlenbruch G, Mahnken AH, et al. Optimized image reconstruction for detection of deep venous thrombosis at multidetector-row CT venography. Eur Radiol 2006;16:269–275.[CrossRef][Medline]
  28. Greco F, Zanolini A, Bova C, et al. Combined diagnostic approach to venous thromboembolism with multidetector computed tomography [in Italian]. Ital Heart J Suppl 2003;4:226–231.[Medline]
  29. Lafitte F, Boukobza M, Guichard JP, Reizine D, Woimant F, Merland JJ. Deep cerebral venous thrombosis: imaging in eight cases. Neuroradiology 1999;41:410–418.[CrossRef][Medline]
  30. Will D, Crayen E, Kirch W, Bruhn HD, Gremmel H, Niedermayer W. Indirect azygos-vein-continuation syndrome with aneurysm of the inferior vena cava [in German]. Klin Wochenschr 1991;69:330–333.[CrossRef][Medline]
  31. Chen JK. Imaging diagnosis of Budd-Chiari syndrome: report of 24 cases [in Chinese]. Zhonghua Yi Xue Za Zhi 1993;73:664–666.[Medline]
  32. Staunton M. Evidence-based radiology: steps 1 and 2—asking answerable questions and searching for evidence. Radiology 2007;242:23–31.[Abstract/Free Full Text]
  33. Malone DE. Evidence-based practice in radiology: an introduction to the series. Radiology 2007;242:12–14.[Free Full Text]
  34. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh, Scotland: Churchill Livingstone, 2000.
  35. Haynes RB. Of studies, syntheses, synopses, and systems: the "4S " evolution of services for finding current best evidence [editorial]. ACP J Club 2001;134:A11–A13.[Medline]
  36. Clinical Evidence Web site. BMJ Publishing Group. http://www.clinicalevidence.com/ceweb/conditions/index.jsp. Accessed October 10, 2005.
  37. American College of Physicians Journal Club Web site. http://www.acpjc.org. Accessed October 10, 2005.
  38. Cochrane Collaboration Web site. http://www.cochrane.org. Accessed October 10, 2005.
  39. Staunton M, Dodd JD, McCormick PA, Malone DE. Finding evidence-based answers to practical questions in radiology: which patients with inoperable hepatocellular carcinoma will survive longer after transarterial chemoembolization? Radiology 2005;237:404–413.[Abstract/Free Full Text]
  40. Maher MM, McNamara AM, MacEneaney PM, Sheehan SJ, Malone DE. Abdominal aortic aneurysms: elective endovascular repair versus conventional surgery—evaluation with evidence-based medicine techniques. Radiology 2003;228:647–658.[Abstract/Free Full Text]
  41. Dodd JD, MacEneaney PM, Malone DE. Evidence-based radiology: how to quickly assess the validity and strength of publications in the diagnostic radiology literature. Eur Radiol 2004;14:915–922.[CrossRef][Medline]
  42. Levels of evidence. Oxford Centre for Evidence-Based Medicine Web site. http://www.cebm.net/levels_of_evidence.asp. Accessed October 10, 2005.
  43. Loud PA, Katz DS, Klippenstein DL, Shah RD, Grossman ZD. Combined CT venography and pulmonary angiography in suspected thromboembolic disease: diagnostic accuracy for deep venous evaluation. AJR Am J Roentgenol 2000;174:61–65.[Abstract/Free Full Text]
  44. Katz DS, Loud PA, Bruce D, et al. Combined CT venography and pulmonary angiography: a comprehensive review. RadioGraphics 2002;22(Spec Issue):S3–S19.[Abstract/Free Full Text]
  45. Peterson DA, Kazerooni EA, Wakefield TW, et al. Computed tomographic venography is specific but not sensitive for diagnosis of acute lower-extremity deep venous thrombosis in patients with suspected pulmonary embolus. J Vasc Surg 2001;34:798–804.[CrossRef][Medline]
  46. Black WC. How to evaluate the radiology literature. AJR Am J Roentgenol 1990;154:17–22.[Free Full Text]
  47. MacEneaney PM, Malone DE. The meaning of diagnostic test results: a spreadsheet for swift data analysis. Clin Radiol 2000;55:227–235.[CrossRef][Medline]
  48. Kline JA, Johns KL, Colucciello SA, Israel EG. New diagnostic tests for pulmonary embolism. Ann Emerg Med 2000;35:168–180.[CrossRef][Medline]
  49. Loud PA, Katz DS, Bruce DA, Klippenstein DL, Grossman ZD. Deep venous thrombosis with suspected pulmonary embolism: detection with combined CT venography and pulmonary angiography. Radiology 2001;219:498–502.[Abstract/Free Full Text]
  50. Lim KE, Hsu YY, Hsu WC, Huang CC. Combined computed tomography venography and pulmonary angiography for the diagnosis PE and DVT in the ED. Am J Emerg Med 2004;22:301–306.[CrossRef][Medline]
  51. Au VW, Walsh G, Fon G. Computed tomography pulmonary angiography with pelvic venography in the evaluation of thrombo-embolic disease. Australas Radiol 2001;45:141–145.[CrossRef][Medline]
  52. Ghaye B, Szapiro D, Willems V, Dondelinger RF. Combined CT venography of the lower limbs and spiral CT angiography of pulmonary arteries in acute pulmonary embolism: preliminary results of a prospective study. JBR-BTR 2000;83:271–278.
  53. Walsh G, Redmond S. Does addition of CT pelvic venography to CT pulmonary angiography protocols contribute to the diagnosis of thromboembolic disease? Clin Radiol 2002;57:462–465.[CrossRef][Medline]
  54. Duwe KM, Shiau M, Budorick NE, Austin JH, Berkmen YM. Evaluation of the lower extremity veins in patients with suspected pulmonary embolism: a retrospective comparison of helical CT venography and sonography. AJR Am J Roentgenol 2000;175:1525–1531.[Abstract/Free Full Text]
  55. Garg K, Kemp JL, Wojcik D, et al. Thromboembolic disease: comparison of combined CT pulmonary angiography and venography with bilateral leg sonography in 70 patients. AJR Am J Roentgenol 2000;175:997–1001.[Abstract/Free Full Text]
  56. Coche EE, Hamoir XL, Hammer FD, Hainaut P, Goffette PP. Using dual-detector helical CT angiography to detect deep venous thrombosis in patients with suspicion of pulmonary embolism: diagnostic value and additional findings. AJR Am J Roentgenol 2001;176:1035–1039.[Abstract/Free Full Text]
  57. Quiroz R, Kucher N, Zou KH, et al. Clinical validity of a negative computed tomography scan in patients with suspected pulmonary embolism: a systematic review. JAMA 2005;293:2012–2017.[Abstract/Free Full Text]
  58. Dodd J, Skehan S, Collins C, et al. An evidence-based radiology algorithmic approach to the diagnosis of venous thromboembolism [abstr]. Eur Radiol 2003;B-0058.
  59. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991;11:88–94.[Abstract/Free Full Text]
  60. Katz DS, Loud PA, Hurewitz AN, Mueller R, Grossman ZD. CT venography in suspected pulmonary thromboembolism. Semin Ultrasound CT MR 2004;25:67–80.[CrossRef][Medline]
  61. Ghaye B, Szapiro D, Willems V, Dondelinger RF. Pitfalls in CT venography of lower limbs and abdominal veins. AJR Am J Roentgenol 2002;178:1465–1471.[Free Full Text]
  62. Abdelmoumene Y, Chevallier P, Barghouth G, et al. Optimization of multidetector CT venography performed with elastic stockings on patients' lower extremities: a preliminary study of nonthrombosed veins. AJR Am J Roentgenol 2003;180:1093–1094.[Free Full Text]
  63. Brink JA. Contrast optimization and scan timing for single and multidetector-row computed tomography. J Comput Assist Tomogr 2003;27(suppl 1):S3–S8.[Medline]
  64. Rademaker J, Griesshaber V, Hidajat N, Oestmann JW, Felix R. Combined CT pulmonary angiography and venography for diagnosis of pulmonary embolism and deep vein thrombosis: radiation dose. J Thorac Imaging 2001;16:297–299.[CrossRef][Medline]
  65. Kalra MK, Maher MM, Toth TL, et al. Strategies for CT radiation dose optimization. Radiology 2004;230:619–628.[Abstract/Free Full Text]
  66. Mayo JR, Aldrich J, Muller NL. Radiation exposure at chest CT: a statement of the Fleischner Society. Radiology 2003;228:15–21.[Abstract/Free Full Text]
  67. Stein PD, Kayali F, Olson RE. Trends in the use of diagnostic imaging in patients hospitalized with acute pulmonary embolism. Am J Cardiol 2004;93:1316–1317.[CrossRef][Medline]
  68. Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative. Radiology 2003;226:24–28.[Abstract/Free Full Text]
  69. Schmidt N, Rutjes AW, van der Windt DA, et al. Quality of reporting of diagnostic accuracy studies. Radiology 2005;235:347–353.[Abstract/Free Full Text]
  70. American College of Radiology Web sit