|
|
||||||||
Thoracic Imaging |
1 From the Department of Radiology, Kinki University School of Medicine, 3772 Oono-higashi, Osaka-Sayama City, Osaka 589-8511, Japan (K.A., Y.N.); Computational Science and Engineering Center, Fujitsu, Chiba, Japan (K.M., A.O.); Department of Radiology, Rinku General Medical Center, Osaka, Japan (M.K., H.H.); and Image Guided Therapy Clinic, Osaka, Japan (S.H.). Received January 13, 2003; revision requested March 20; final revision received July 7; accepted July 17. Address correspondence to K.A.
| ABSTRACT |
|---|
|
|
|---|
MATERIALS AND METHODS: Fifty chest computed tomographic (CT) examination cases were used. The mean nodule size was 0.81 cm ± 0.60 (SD) (range, 0.32.9 cm). Alternative free-response receiver operating characteristic (ROC) analysis with a continuous rating scale was used to compare the observers performance in detecting nodules with and without use of CAD. Five board-certified radiologists and five radiology residents participated in an observer performance study. First they were asked to rate the probability of nodule presence without using CAD; then they were asked to rate the probability of nodule presence by using CAD.
RESULTS: For all radiologists, the mean areas under the best-fit alternative free-response ROC curves (Az) without and with CAD were 0.64 ± 0.08 and 0.67 ± 0.09, respectively, indicating a significant difference (P < .01). For the five board-certified radiologists, the mean Az values without and with CAD were 0.63 ± 0.08 and 0.66 ± 0.09, respectively, indicating a significant difference (P < .01). For the five resident radiologists, the mean Az values without and with CAD were 0.66 ± 0.04 and 0.68 ± 0.04, respectively, indicating a significant difference (P = .02). At observer performance analyses, there were no significant differences in Az values obtained either without (P = .61) or with (P = .88) CAD between the board-certified radiologists and the residents. For all radiologists, in the detection of pulmonary nodules 1.0 cm in diameter or smaller, the mean Az values without and with CAD were 0.60 ± 0.11 and 0.64 ± 0.11, respectively, indicating a significant difference (P < .01).
CONCLUSION: Use of the CAD system improved the board-certified radiologists and residents detection of pulmonary nodules at chest CT.
© RSNA, 2004
Index terms: Computers, diagnostic aid Diagnostic radiology, observer performance Lung, CT, 60.12111, 60.12115 Lung, nodule, 60.281
| INTRODUCTION |
|---|
|
|
|---|
In the screening for lung cancer with CT, however, radiologists have to analyze large amounts of data, numerous image sections per case, and 50100 cases per day. There is always the risk of missing a lesion. In a retrospective study of first annual CT examinations, Swensen et al (7) found that nodules were missed in 26% of patients. There are some methods to help avoid missing a pulmonary nodule, such as independent reading by two or more radiologists and the use of computer-aided diagnosis (CAD) for the detection of pulmonary nodules. Some researchers have reported the use of a CAD system in lung cancer screening with CT (918). We also have developed an integrated CAD system for lung cancer screening with CT (19). The purpose of the present study was to evaluate the effect of using a CAD system on radiologists performance in detecting pulmonary nodules.
| MATERIALS AND METHODS |
|---|
|
|
|---|
All chest CT examinations were performed by using a LightSpeed QX/i scanner (GE Medical Systems, Milwaukee, Wis) without contrast material administration. Helical CT images of the entire lung were obtained by using a detector row width of 5.0 mm, helical pitch of 3.0, 7.5-mm section thickness and intervals, 0.8-second rotation time, 120 kVp, and 160200 mA. Fifty-one of the 133 cases (of 133 patients) were excluded because of the presence of four or more pulmonary nodules, a pulmonary nodule larger than 3 cm in diameter, severe pulmonary fibrosis, diffuse bronchiectasis, or extensive inflammatory scars.
The chest CT images in the remaining 82 cases (of 82 patients) were reviewed for the location, number, and size of pulmonary nodules. The CT images obtained in the 133 cases were reviewed by two experienced radiologists (K.A. and S.H.) who did not participate in the observer performance study. K.A. and S.H. had more than 16 and 27 years of CT imaging experience, respectively. The two radiologists reviewed all images twice, with an interval of 1 month between the two review sessions. A final interpretation was performed by consensus.
The 82 cases were those of 29 patients without pulmonary nodules, 33 patients with one nodule, 15 patients with two nodules, and five patients with three nodules. The nodules examined in this study were located both centrally and peripherally in the lung parenchyma. In general, it is difficult to identify nodules smaller than 0.3 cm on 7.5-mm-thick CT image sections. Therefore, the two radiologists did not search for nodules smaller than 0.3 cm. As a result, the smallest nodules included in this study were 0.3 cm. The mean size of the 78 nodules was 0.89 cm ± 0.67 (SD) (range, 0.33.0 cm). Fifty-five nodules were 0.31.0 cm in diameter, and 23 nodules were 1.13.0 cm in diameter. A functional evaluation of our CAD system was performed by using these 82 cases.
Before the observer performance study, a pilot study was conducted with two observers, who were not involved in the observer performance study. It took these observers about 5 hours to read the images obtained in all 82 cases in the pilot study; this indicated that fatigue owing to reading would be an important factor in the observer performance study. For this reason, 50 of the 82 cases were randomly chosen for the observer performance study. These 50 cases were those of 14 patients without pulmonary nodules, 21 patients with one nodule, 10 patients with two nodules, and five patients with three nodules. The mean size of these 56 nodules was 0.81 cm ± 0.60 (range, 0.32.9 cm). Forty-five of these nodules were 0.31.0 cm in diameter, and 11 nodules were 1.12.9 cm in diameter. These 50 cases were those of 27 men and 23 women aged 2881 years (mean age, 57.8 years). The mean age of the men was 56 years (age range, 2881 years), and the mean age of the women was 60 years (age range, 4273 years).
Computerized Scheme for Automated Detection of Pulmonary Nodules
Our method of nodule detection at CT is outlined in Figure 1. First, the lungs were segmented by using a gray-level threshold (-300 HU). The gray level selected for lung segmentation has resulted in the erroneous exclusion of nodules, vessels, and bronchi within the lungs. To compensate for this type of segmentation error, a three-dimensional labeling technique (20) and a mathematic morphologic technique (21) were used. Second, intrapulmonary structures such as pulmonary nodules, pulmonary vessels, and bronchi were segmented by using the top-hat transformation technique (22), with which a smoothed image is subtracted from the original image. Third, initial potential nodules were identified by using a sieve filter, which is used to analyze the intrapulmonary structures with the skeleton technique (22) and to select structures that are larger than a predefined size. The mesh size of the sieve filter is altered according to the section level in the vertical direction and the location within the lungs on individual sections.
|
A workstation (Windows NT, PRIMERGY, with Dual Intel Pentium III [1.0-GHz] processors; Fujitsu, Tokyo, Japan) was used in this study, and the average nodule detection time for each case was about 7 minutes. To indicate the CAD output, the computer-detected location of each pulmonary nodule was marked by a small square, the center of which was placed in the center of the nodule. The computer-detected locations of potential nodules were marked by small squares. The true nodule was indicated by a large square. Figure 2 shows an example of this automatic nodule detection at chest CT.
|
The results of automated detection of pulmonary nodules performed by using the NT workstation were stored in the database of a picture archiving and communication system, or PACS (HOPE/Dr ABLE-Ex; Fujitsu, Tokyo, Japan). The 10 observers read the CT images displayed on a gray-scale monitor (model SMM21200P; Siemens, Munich, Germany) with a spatial resolution of 2,048 x 2,560. The monitor screen could be split into up to 24 parts to display the CT images, and the observers were allowed to select the number of split parts on the monitor screen. They were also permitted to use the cine mode for displaying the images.
All 50 cases (of 50 patients) were presented to the observers in the same order. The observers were provided with the following information before the test to expedite the observer performance study: (a) The sequential test method was being used, (b) the cases of patients with nodules accounted for about 70% of all the cases, (c) the number of nodules was three or fewer in each case, and (d) the nodules were 0.33.0 cm in diameter. No restriction was placed on reading time.
Each observer used a continuous rating scale of a line-marking method to rate his or her confidence level by marking on a line that was 7 cm in length. The left end of the line indicated complete confidence that the chest CT image showed no nodule, whereas the right end indicated complete confidence that the chest CT image showed a nodule. Intermediate levels of confidence were indicated by the different positions of the marks between these two ends on the line, and positions close to the right and left ends indicated, respectively, greater and lesser degrees of confidence regarding the presence of a nodule. One author (H.H.) then measured the distance between the left end and the marked point and converted this distance to an ordinal confidence rating that ranged from 0 to 100.
The results of the initial review by the two radiologists (K.A., S.H.) who did not participate in the observer study were used as the reference standard. A continuous rating scale containing a pair of horizontal lines was used in the sequential test. Observers first recorded their noncomputer-aided rating results on the upper line; then, they rerecorded their rating results on the lower line after seeing the CAD output. They entered the reading results for each case on a record form. This form had six sets of continuous rating scales containing two horizontal lines each (two sets each for up to three possible nodules). Each observer was required to mark the continuous rating scale to indicate his or her level of confidence regarding the presence or absence of a nodule in each case and to record the number of the section showing a nodule and the general schematic location of the nodule on the right side. Before the observer test, each observer underwent a training session that involved reading the images obtained in five training cases to become familiarized with the observer test. These five training cases were not a part of the 50 cases used in the observer performance study.
Statistical Analysis
Observer performance was evaluated by using alternative free-response receiver operating characteristic (ROC) analysis, in which one takes into account nodule location and which allows evaluation of multiple nodules per case (24,25). Alternative free-response ROC curves for each observer when not using and when using the CAD output were calculated by plotting the true-positive fraction against the likelihood of obtaining an image with false-positive findings (ie, with one or more false-positive lesions) at each confidence level. The area under each alternative free-response ROC curve (Az) was used to compare the observers performance in detecting pulmonary nodules when they did not use CAD with their performance when they did use CAD. Analyses of the detection of all nodules and of the detection of nodules 1 cm in diameter or smaller were performed.
The significance of the difference between the Az values obtained without and those obtained with CAD outputs was evaluated with a two-tailed paired t test. The significance of the difference in Az values between the board-certified radiologists and the radiology residents was evaluated with a two-tailed two-sample t test. P values of less than .05 were considered to indicate a significant difference. Statistical analyses were performed by using a statistical software package (StatView, version 5.0; SAS Institute, Cary, NC).
| RESULTS |
|---|
|
|
|---|
The graph in Figure 3 shows the averaged alternative free-response ROC curves for all 10 observers in the detection of all pulmonary nodules. The Az values for all 10 observers in the detection of all pulmonary nodules without and with CAD output are summarized in Table 1. The mean Az values obtained without and with CAD output were 0.64 ± 0.08 and 0.67 ± 0.09, respectively, indicating a significant difference (P < .01).
|
|
|
|
|
|
| DISCUSSION |
|---|
|
|
|---|
A true-positive rate of 100%, a false-negative rate of 0%, and a false-positive rate of 0 nodule per section would be ideal. In fact, however, as the true-positive rate approaches 100%, the false-positive rate also tends to increase. Therefore, to develop a clinically useful automatic system for the detection of pulmonary nodules, an increased true-positive rate and decreased false-positive rate, with a sufficient balance between the two values, are necessary. With present technical levels, the developmental target of our CAD system is a true-positive rate of 90% or greater and a false-positive rate of 0.1 or fewer nodule per section. The reformation of our CAD system is ongoing.
The results of the observer study suggest that the use of our CAD system led to improved performance in the detection of pulmonary nodules at chest CT for both the radiology residents and the board-certified radiologists. There was no significant difference between the two groups in their performance in detecting pulmonary nodules either without or with CAD output. In the observer study, the observers were required only to detect pulmonary nodules: They were not required to determine the likelihood of malignancy of any lesions.
The basic knowledge required to detect pulmonary nodules is only the sectional anatomy of the lungs. All of the residents who took part in the observer study had 2 or more years of experience and were considered to be thoroughly familiar with the sectional anatomy of the lungs, like the board-certified radiologists. This factor presumably accounts for the finding that there was no significant difference between the two groups in their performance in detecting pulmonary nodules either without or with CAD output. In other words, radiologists performance in detecting pulmonary nodules probably depends more on how attentively each observer reads CT images than on his or her experience and knowledge as a radiologist (with the exclusion of their knowledge of the sectional anatomy). In these conditions, the use of CAD is expected to improve radiologists detection performance, irrespective of the knowledge or experience of each observer.
The Az values for all the observers in the detection of nodules 1 cm in diameter or smaller were significantly higher with than without CAD output. However, for one board-certified radiologist and one radiology resident, the Az values for the detection of these small nodules were lower with than without CAD output. In general, it is frequently difficult to distinguish a 1-cm or smaller nodule from the cross section of a blood vessel on an image with a relatively large (ie, 7.510.0-mm) section thickness. Reading on a workstation in cine mode is useful for tracing the continuity of a blood vessel (26). However, even in cine mode, it is frequently difficult to distinguish a small nodule of about 5 mm from a blood vessel on an image with a section thickness of 7.510.0 mm.
The present problem with our CAD system is a high false-positive rate. When a nodule is smallthat is, 1 cm or smallerit is difficult to verify in a short time the presence of such a small nodule that has been identified by using the CAD system. This appears to be the reason that the Az values were lower with than without CAD output for two observers. Use of an image with a thin section thicknessthat is, 12 mmmakes it easier to distinguish a nodule 1 cm or smaller from a blood vessel. However, our system was developed for use with relatively thick sectionsthat is, 510 mm. Therefore, it is necessary to develop a new system with which CAD can be used with images with section thicknesses of 12 mm.
The minimum target size of a nodule at CT lung cancer screening is important for setting scanning parameters (ie, section thickness, section interval, detector row width, helical pitch, and reconstruction algorithm) and determining the detection capacity of the CAD system. The minimum target size of a nodule must be decided with consideration of how much improvement in prognosis is sought, after confirming the correlation between the pulmonary nodule size and the prognosis. Although some study results suggest that pulmonary nodule size and prognosis do not necessarily correlate (27), the results of a study by Sobue et al (8) suggest that small lung cancers are associated with a better survival rate. According to the results of that study, the 5-year survival rate was almost 100% for patients with nodules 9 mm or smaller. However, they considered all nodules 9 mm or smaller in their analysis and did not include a breakdown of the 5-year survival rates for patients with pulmonary nodules 9 mm or smaller. Although we developed our CAD system with the aim of detecting nodules up to 3 mm in diameter out of convenience, more studies are needed to determine the actual minimum target size.
This study had several limitations. First, to expedite the observer performance study, we provided the observers with certain information, such as the number of nodules in each case, the sizes of the nodules, and the fact that the cases with nodules accounted for about 70% of the total number of cases. However, providing such information may have skewed the observational data. In daily clinical work, more nodules may actually be present. Furthermore, if the readers assume that only three nodules are present, they will stop searching for nodules after identifying three, even though they perceive that there are more, and, thus, the potentially false-positive reports will be artificially excluded.
Second, we excluded the cases of severe pulmonary fibrosis, diffuse bronchiectasis, and extensive inflammatory scars because lung segmentation with the CAD system may be difficult in cases with such severe interstitial lung disease. We also excluded the patients who had four or more pulmonary nodules and those who had pulmonary nodules larger than 3 cm in diameter. However, these exclusions may have biased the results in favor of the usefulness of the CAD system.
Third, we used the sequential test method in the observer performance study. However, reading the images without CAD output and then reading them with CAD output may have introduced a training effect.
Fourth, the observers were allowed to select the number of split parts on the monitor screen in the observer performance study; however, there might be a substantial difference in the conspicuity of nodules when the screen is one part or split into 24 parts to display CT images. This factor may partially explain the poorer performance of the radiologists as compared with that of the CAD system.
Fifth, we used the results of the initial review by the two experienced radiologists as a reference standard in the observer performance study; however, determining whether they accurately identified the "true" nodules is problematic. Pathologic confirmation for or clinical follow-up of patients to assess nodule growth patterns is necessary to identify and avoid missing the true nodules.
In conclusion, the use of our CAD system helped to improve both the residents and the board-certified radiologists performance in detecting pulmonary nodules. However, the next challenge is to decrease the false-positive rate associated with our CAD system.
| FOOTNOTES |
|---|
Author contributions: Guarantors of integrity of entire study, K.A., K.M., S.H.; study concepts, K.A., S.H.; study design, K.A., K.M.; literature research, K.A.; clinical studies, K.A., M.K., H.H., S.H.; data acquisition, K.A., M.K., H.H.; data analysis/interpretation, K.A., M.K., H.H., S.H.; statistical analysis, H.H., M.K.; manuscript preparation and definition of intellectual content, K.A., M.K.; manuscript editing, K.A., Y.N., S.H.; manuscript revision/review, K.A., Y.N.; manuscript final version approval, all authors
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
N. Kawel, B. Seifert, M. Luetolf, and T. Boehm Effect of Slab Thickness on the CT Detection of Pulmonary Nodules: Use of Sliding Thin-Slab Maximum Intensity Projection and Volume Rendering Am. J. Roentgenol., May 1, 2009; 192(5): 1324 - 1329. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. Girvin and J. P. Ko Pulmonary Nodules: Detection, Assessment, and CAD Am. J. Roentgenol., October 1, 2008; 191(4): 1057 - 1069. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Hock, R. Ouhadi, R. Materne, A.-S. Aouchria, I. Mancini, T. Broussaud, P. Magotteaux, and A. Nchimi Virtual Dissection CT Colonography: Evaluation of Learning Curves and Reading Times with and without Computer-aided Detection Radiology, September 1, 2008; 248(3): 860 - 868. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. Robinson, S. Halligan, S. A. Taylor, S. Mallett, and D. G. Altman CT Colonography: A Systematic Review of Standard of Reporting for Studies of Computer-aided Detection Radiology, February 1, 2008; 246(2): 426 - 433. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. Beigelman-Aubry, P. Raffy, W. Yang, R. A. Castellino, and P. A. Grenier Computer-Aided Detection of Solid Lung Nodules on Follow-Up MDCT Screening: Evaluation of Detection, Tracking, and Reading Time Am. J. Roentgenol., October 1, 2007; 189(4): 948 - 955. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. E. Baker, L. Bogoni, N. A. Obuchowski, C. Dass, R. M. Kendzierski, E. M. Remer, D. M. Einstein, P. Cathier, A. Jerebko, S. Lakare, et al. Computer-aided Detection of Colorectal Polyps: Can It Improve Sensitivity of Less-Experienced Readers? Preliminary Findings Radiology, October 1, 2007; 245(1): 140 - 149. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. Ikeda, K. Awai, T. Mori, K. Kawanaka, Y. Yamashita, and H. Nomori Differential Diagnosis of Ground-Glass Opacity Nodules: CT Number Analysis by Three-Dimensional Computerized Quantification Chest, September 1, 2007; 132(3): 984 - 990. [Abstract] [Full Text] [PDF] |
||||
![]() |
H Bolte, C Riede, S Muller-Hulsbeck, S Freitag-Wolf, G Kohl, T Drews, M Heller, and J Bieder Precision of computer-aided volumetry of artificial small solid pulmonary nodules in ex vivo porcine lungs Br. J. Radiol., June 1, 2007; 80(954): 414 - 421. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. D. O'Connor, J. Yao, and R. M. Summers Lytic Metastases in Thoracolumbar Spine: Computer-aided Detection at CT--Preliminary Study Radiology, March 1, 2007; 242(3): 811 - 816. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Das, G. Muhlenbruch, A. H. Mahnken, T. G. Flohr, L. Gundel, S. Stanzel, T. Kraus, R. W. Gunther, and J. E. Wildberger Small Pulmonary Nodules: Effect of Two Computer-aided Detection Systems on Radiologist Performance. Radiology, November 1, 2006; 241(2): 564 - 571. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. D. O'Connor, R. M. Summers, J. Yao, P. J. Pickhardt, and J. R. Choi CT Colonography with Computer-aided Polyp Detection: Volume and Attenuation Thresholds to Reduce False-Positive Findings Owing to the Ileocecal Valve Radiology, November 1, 2006; 241(2): 426 - 432. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. Yuan, P. M. Vos, and P. L. Cooperberg Computer-Aided Detection in Screening CT for Pulmonary Nodules. Am. J. Roentgenol., May 1, 2006; 186(5): 1280 - 1287. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. Awai, K. Murao, A. Ozawa, Y. Nakayama, T. Nakaura, D. Liu, K. Kawanaka, Y. Funama, S. Morishita, and Y. Yamashita Pulmonary Nodules: Estimation of Malignancy at Thin-Section Helical CT--Effect of Computer-aided Diagnosis on Performance of Radiologists Radiology, April 1, 2006; 239(1): 276 - 284. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Gur Lung Cancer Screening: Radiology's Opportunity Here and Now Radiology, February 1, 2006; 238(2): 395 - 397. [Full Text] [PDF] |
||||
![]() |
F. Li, H. Arimura, K. Suzuki, J. Shiraishi, Q. Li, H. Abe, R. Engelmann, S. Sone, H. MacMahon, and K. Doi Computer-aided Detection of Peripheral Lung Cancers Missed at CT: ROC Analyses without and with Localization Radiology, November 1, 2005; 237(2): 684 - 690. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. Hirai, Y. Korogi, H. Arimura, S. Katsuragawa, M. Kitajima, M. Yamura, Y. Yamashita, and K. Doi Intracranial Aneurysms at MR Angiography: Effect of Computer-aided Diagnosis on Radiologists' Detection Performance Radiology, November 1, 2005; 237(2): 605 - 610. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. G. Kim, J. M. Goo, J. H. Kim, H. J. Lee, B. G. Min, K. T. Bae, and J.-G. Im Computer-aided Diagnosis of Localized Ground-Glass Opacity in the Lung at CT: Initial Experience Radiology, November 1, 2005; 237(2): 657 - 661. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. K. Leader, T. E. Warfel, C. R. Fuhrman, S. K. Golla, J. L. Weissfeld, R. S. Avila, W. D. Turner, and B. Zheng Pulmonary Nodule Detection with Low-Dose CT of the Lung: Agreement Among Radiologists Am. J. Roentgenol., October 1, 2005; 185(4): 973 - 978. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Novello, C. Fava, P. Borasio, L. Dogliotti, G. Cortese, B. Crida, G. Selvaggi, P. Lausi, M. P. Brizzi, P. Sperone, et al. Three-year findings of an early lung cancer detection feasibility study with low-dose spiral computed tomography in heavy smokers Ann. Onc., October 1, 2005; 16(10): 1662 - 1666. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. P. Ko, J. Chang, E. Bomsztyk, J. S. Babb, D. P. Naidich, and H. Rusinek Effect of CT Image Compression on Computer-assisted Lung Nodule Volume Measurement Radiology, October 1, 2005; 237(1): 83 - 88. [Abstract] [Full Text] [PDF] |
||||
![]() |
K Doi Current status and future potential of computer-aided diagnosis in medical imaging Br. J. Radiol., January 1, 2005; 78(suppl_1): S3 - s19. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |