|
|
||||||||
Breast Imaging |
1 From the Divisions of Breast Imaging and Intervention (M.J.M., D.H.W., K.R.B., K.K.A.) and Biostatistics (J.N.M.), Mayo Clinic, 200 First St SW, Rochester, MN 55905. Received December 14, 2004; revision requested February 4, 2005; revision received May 12; accepted June 13; final version accepted July 1. Address correspondence to M.J.M. (e-mail: morton.marilyn{at}mayo.edu).
| ABSTRACT |
|---|
|
|
|---|
Materials and Methods: Institutional review board approval was granted; informed consent and HIPAA compliance were waived. A total of 21 349 screening mammograms obtained in 18 096 women were interpreted first without and then with review of CAD images to determine the effect of CAD analysis on the screening breast cancer detection rate, recall rate, and positive predictive value (PPV) for biopsy. The percentage of total cancers detected by the radiologists independent of CAD and the percentage correctly marked by the CAD system were determined.
Results: On the basis of pre-CAD interpretations, 2101 patients were recalled for diagnostic evaluation, 256 biopsies were performed, and 105 breast cancers were diagnosed. The breast cancer detection rate per 1000 screening mammograms was 4.92 (105 of 21 349 mammograms), the recall rate was 9.84% (2101 of 21 349 mammograms), and the PPV for biopsy was 41.0% (105 of 256 biopsies). After CAD image review, 199 additional patients were recalled, 21 additional biopsies were performed, and eight additional cancers were detected. The effect was a 7.62% (eight of 105) increase in the number of breast cancers detected, an increase in the recall rate to 10.77% (2300 of 21 349 mammograms), and a slight decrease in the PPV to 40.8% (113 of 277 biopsies). Radiologists detected 92.9% (105 of 113 cancers) of the total cancers, and CAD correctly marked 76.1% (86 of 113 cancers).
Conclusion: The use of CAD improved the detection of breast cancer, with an acceptable increase in the recall rate and a minimal increase in the number of biopsies with benign results.
© RSNA, 2006
| INTRODUCTION |
|---|
|
|
|---|
Several studies have shown that nonpalpable breast cancers detected at screening are often visible in retrospect on prior mammograms. In studies in which a nonblinded retrospective review was performed (ie, the current mammogram that led to a diagnosis of breast cancer was compared with the prior mammogram), up to 77% of breast cancers were visible on the prior mammogram (6,10,1214). By comparison, studies that involved blinded review of prior mammograms found that 25%41% of screening-detected breast cancers were visible retrospectively (7,10,14). Although the specific reason that a given breast cancer is overlooked at mammographic interpretation is not always readily apparent, many undetected breast cancers are overlooked because of perceptual errors on the part of the interpreting radiologist (8,10,15,16).
To reduce perceptual errors, independent double reading of screening mammograms by two or more radiologists has been investigated. Double reading increased the number of screening-detected breast cancers by 5%15% (1720). Although commonly performed in European countries, this method of double interpretation is not widely practiced in the United States, owing to the increased resources required for implementation.
To date, most studies in which an evaluation of the potential usefulness of computer-aided detection (CAD) as an aid to radiologists in their interpretation of mammograms has been performed have been retrospective. Several investigators have evaluated the potential of CAD to reduce the false-negative interpretations by performing CAD analysis on prior mammograms of patients with screening-detected breast cancers. On the basis of the ability of the CAD system to mark retrospectively visible cancers, investigators in two similarly designed studies (14,21) estimated the potential benefit of CAD to be an increase in the breast cancer detection rate of 21%. Although these results indicate that CAD systems do have the potential to increase the mammographic detection of early-stage breast cancer, the researchers in these studies did not directly measure the effect of CAD as an interpretation aid in actual clinical practice. To our knowledge, the investigators in only one published study (22) have evaluated the effects of CAD on the interpretation of screening mammograms in a clinical practice setting. In this prospective study, CAD analysis resulted in a 19.5% increase in the breast cancer detection rate.
The purpose of our study, thus, was to determine prospectively the effect of a commercially available CAD system on interpretations of screening mammograms.
| MATERIALS AND METHODS |
|---|
|
|
|---|
Image Acquisition and CAD Analysis
Each screen-film mammogram was acquired with one of four identical screen-film mammographic systems (Lorad MIV; Hologic Systems, Bedford, Mass) by a certified mammographic technologist in accordance with the rules and regulations set forth by the Mammography Quality Standards Act of 1992 (23). The CAD analysis was performed with a commercially available CAD system (ImageChecker M1000, version 2.2; R2 Technology, Sunnyvale, Calif) composed of a film digitizer to convert the analog films to digital images, a processing computer for application of the CAD algorithms, and an image display system. CAD images were displayed on low-resolution video monitors built into a motorized film viewer (RADX 614A; S&S Technology, Houston, Tex). The CAD system algorithms use a triangle to mark clusters of bright spots (suggestive of clustered microcalcifications) and an asterisk to mark areas of central density and radiating lines (suggestive of a mass or architectural distortion). With use of a continuous-loading digitizer, a routine four-view mammogram was digitized and processed in approximately 4 minutes.
Interpretation of Mammograms
Each screening mammogram was independently interpreted by one of 12 board-certified radiologists (including M.J.M., D.H.W., K.R.B., and K.K.A.), each of whom was aware of his or her participation in the study. The range of experience in mammographic interpretation was 330 years, with a mean of 12 years. In each case, the radiologist made an initial interpretation and assessment without viewing the CAD images. The initial assessment was either negative or benign (Breast Imaging Reporting and Data System 1 or 2) (24) or incomplete (Breast Imaging Reporting and Data System 0) (24). An incomplete assessment generated a recall of the patient for diagnostic imaging or a request to obtain and compare prior mammograms from other facilities, or both. The CAD images were then viewed on video display monitors, and the regions of interest marked by the CAD system were reexamined on the hard-copy film images. An abnormal mammographic finding detected by the radiologist was considered to be similarly detected with CAD if the appropriate CAD mark was present on at least one of the two views of each breast. If the potential abnormality had features of both a mass and microcalcifications, then the presence of either CAD mark, asterisk, or triangle was considered correct. The absence of a CAD mark on a potential abnormality did not alter the radiologist's initial decision to recall the patient on the basis of the pre-CAD interpretation. A final assessment and a recommendation were made after the completion of the diagnostic evaluation.
Data Collection
For each mammographic finding that resulted in the recall of a patient for additional diagnostic imaging (Breast Imaging Reporting and Data System 0), the following data were recorded by the interpreting radiologist: (a) for the method of detection, findings detected at the pre-CAD interpretation and also marked by the CAD system were recorded, findings not detected at the pre-CAD interpretation but marked by the CAD system with subsequent recall of the patient were recorded, and findings detected at the pre-CAD interpretation but not marked by the CAD system were recorded; (b) the predominant mammographic finding (mass or density, architectural distortion, or clustered microcalcifications); (c) the location within the breast (quadrant or clock position); and (d) the distance of the finding from the nipple recorded in centimeters.
The final interpretation assessment and recommendation for each screening mammogram in a patient recalled for diagnostic imaging were retrieved from our mammography database by one of the authors (D.H.W.). All percutaneous and surgical breast biopsy results and breast cancer staging data are maintained within this database. The histopathologic diagnosis of all lesions on which a biopsy was performed and the maximum dimension and stage of all screening-detected breast cancers were recorded by two of the authors (M.J.M. and D.H.W.).
Statistical Analysis
The recorded data were collectively analyzed by three individuals (including M.J.M. and D.H.W.) to determine the effect of CAD on the screening breast cancer detection rate (the number of occult cancers detected per 1000 examinations), the screening recall rate (the number of patients in whom screening mammograms were obtained who were recalled for diagnostic imaging divided by the total number of screening examinations), and the positive predictive value (PPV) for biopsy (the number of cancers diagnosed divided by the total number of biopsies performed).
The number of breast cancers detected by the radiologists independent of CAD analysis and the number correctly marked by the CAD system were calculated as percentages of the total breast cancers detected. For each breast cancer diagnosed that was not initially perceived by the radiologist and for each breast cancer not marked by the CAD system but detected by the radiologist, the specific mammographic finding (mass or density, architectural distortion, or microcalcifications), tumor histologic findings, and maximum tumor dimension were determined. In the absence of a paired control group, 95% confidence intervals were calculated from the recorded data. All analyses were performed with statistical software (SAS, version 8, 2001; SAS Institute, Cary, NC).
| RESULTS |
|---|
|
|
|---|
|
|
After the CAD images were reviewed and the computer-marked regions of interest were reexamined by the radiologist, the an additional 199 patients were recalled. In these patients who were recalled, an additional 21 breast biopsies were performed, and this number resulted in an 8.20% (21 of 256) relative increase in the total number of breast biopsies performed. From these additional biopsies, eight breast cancers were diagnosed that had not been detected at the pre-CAD interpretations. Detection of these eight additional breast cancers increased the breast cancer detection rate from 4.92 to 5.29 (113 of 21 349 mammograms), a relative increase of 7.62% (eight of 105) in the number of breast cancers detected.
The proportion of benign and malignant breast lesions from which tissue samples were removed at biopsy was only slightly affected by the use of CAD. The PPV for the CAD-generated breast biopsies was 38% (eight of 21 biopsies) compared with the PPV of 41.0% (105 of 256 biopsies) for biopsies generated by the radiologists on the basis of their pre-CAD interpretations. The combined total PPV for biopsy was 40.8% (113 of 277 biopsies), a minimal decrease from the pre-CAD PPV of 41.0%.
Mammographic Findings
Among the total 113 breast cancers diagnosed from the 21 349 screening mammographic interpretations, a visible mass or density was the most common finding (72 [63.7%] of 113), followed by clustered microcalcifications (31 [27.4%]) and architectural distortion (10 [8.85%]). The radiologists detected 105 (92.9%) of 113 malignancies: 70 (97%) of 72 malignant masses, nine (90%) of 10 malignant architectural distortions, and 26 (84%) of 31 malignant clustered microcalcifications. The CAD system correctly marked 86 (76.1%) of 113 malignancies: 47 (65%) of 72 malignant masses, eight (80%) of 10 malignant architectural distortions, and all 31 (100%) malignant clustered microcalcifications.
Cancers Not Marked by CAD System
Of the 105 breast cancers detected by the radiologists, 27 (25.7%) were not marked by the CAD system (Table 3). In 25 (93%) of the 27 unmarked cancers, a mass or density was the predominant mammographic finding (Fig 1). These lesions ranged in size from 0.5 to 2.2 cm in greatest diameter. Architectural distortion was the predominant mammographic finding in the remaining two cancers not marked by CAD. Both were invasive ductal carcinomas, not otherwise specified, measuring 0.7 and 1.7 cm in greatest diameter. In none of the cancers overlooked by the CAD system were microcalcifications the predominant mammographic finding.
|
|
|
|
|
|
|
|
| DISCUSSION |
|---|
|
|
|---|
In a retrospective study by Warren Burhenne et al (14), 1083 clinically occult breast cancers were detected from current screening mammograms. In 427 of these cases, the most recent prior mammogram was available for review. In 67% (286 of 427) of these prior mammograms, the breast cancers were found to be visible in retrospect. At independent blinded review, 27% (115 of 427) of the prior mammograms were interpreted as warranting a recall of the patient for diagnostic evaluation. The 1083 current mammograms and each of the 427 prior mammograms were then analyzed with the ImageChecker CAD system. The CAD system correctly marked 84% (906 of 1083) of the current cancers, including 99% (400 of 406) of malignant calcifications and 75% (506 of 677) of cancers manifesting as a mass, asymmetric density, or architectural distortion. The CAD system correctly marked 89 (77%) of 115 retrospectively visible breast cancers considered to warrant a recall of the patient. Warren Burhenne et al (14) determined that the original radiologists had a false-negative rate of 21% and estimated that the use of CAD analysis could have reduced this rate by 77%.
In a more recent retrospective study, Brem et al (21) evaluated the ability of Second Look to correctly mark breast cancers overlooked on prior mammograms. Screening mammograms from which 930 occult breast cancers were detected were identified, and a total of 377 prior mammograms were available for review. In 47% (177 of 377) of the prior mammograms, the breast cancers were considered to be visible in retrospect by a panel of radiologists. These 177 prior screening mammograms were subsequently analyzed with Second Look, and 63% (111 of 177) of the overlooked cancers were correctly marked by the CAD system. In these two retrospective studies, the investigators determined that, if the initial mammographic interpretations had been established with CAD assistance, there could have been a 20% (14) and 21% (21) increase in the breast cancer detection rate.
To our knowledge, the only published prospective study of the use of CAD to supplement screening mammographic interpretations in clinical practice was reported by Freer and Ulissey (22). These authors reported their results with ImageChecker to interpret 12 860 screening mammograms. The radiologists detected 41 cancers (detection rate, 3.2 cancers per 1000 screening mammograms) without CAD assistance. Eight additional cancers, seven of which were clustered microcalcifications, were detected only after the radiologists reviewed the CAD-marked regions of interest. Detection of these additional cancers resulted in an increase in the breast cancer detection rate of 19.5% (eight of 41). This finding is similar to the potential estimated increases in breast cancer detection of 20% and 21% reported in the retrospective studies by Warren Burhenne et al (14) and Brem et al (21).
Freer and Ulissey (22) found that the radiologists detected 84% (41 of 49) of the total number of breast cancers at the pre-CAD interpretations, and the CAD system correctly marked 82% (40 of 49 cancers). In their experience, the effect of CAD analysis on the recall of patients was an absolute increase in the screening recall rate of 1.2% (from 6.5% to 7.7%) and a relative increase of 18.46% (1.2 of 6.5). The PPV for biopsy before and after CAD was 38%. All eight additional cancers detected only after CAD image review were stage 0 and 1 lesions.
To our knowledge, our study is the largest prospective evaluation of the effect of CAD assistance on the interpretation of screening mammograms in clinical practice. From the 21 349 interpretations of screening mammograms, 105 breast cancers were detected (4.9 per 1000 screening mammograms) by the radiologists independent of CAD image review. CAD analysis resulted in the detection of eight additional breast cancers that the radiologists did not detect prospectively. Thus, a total of 113 clinically occult breast cancers (5.3 per 1000 screening mammograms) were detected by the radiologists with CAD assistance. These additional eight CAD-detected cancers represent a relative increase in the breast cancer detection rate of 7.62% (eight of 105). This result differs from the estimated increase in breast cancer detection of 20%21% found in two retrospective studies (14,21), as well as from the 19.5% increase reported in the prospective study by Freer and Ulissey (22).
We found that the radiologists detected 92.9% (105 of 113 cancers) of the total breast cancers without CAD assistance and that the CAD system correctly marked 76.1% (86 of 113) of the total breast cancers. Our results differ somewhat from the sensitivity rates of 84% for the radiologist and 82% for the CAD system reported by Freer and Ulissey (22). A comparison of the various studies about the evaluation of the effect of CAD on screening mammographic interpretations can be found in Table 5. Variations in reported CAD sensitivity rates may be attributable to differences in the specific mammographic features of the screening-detected breast cancers in any given study (microcalcifications vs mass vs architectural distortion). CAD algorithms are known to have a higher sensitivity for marking malignant calcifications than for marking malignant masses (14,15,22,26,27). In addition, CAD sensitivity values have been shown to vary according to the specifications of the CAD software used (28). Breast cancer detection rates and sensitivity values would also be expected to vary according to the expertise of the individual radiologist and to be influenced by differences in the prevalence and incidence rates of breast cancers in the populations screened. A higher prevalence of disease would be expected in our screening patient population (mean age, 60 years) compared with that (mean age, 49 years) in the study reported by Freer and Ulissey (22).
|
The results of studies in which the investigators evaluated the sensitivity of CAD algorithms for detection of malignant masses have been less impressive. The mass detection algorithm of current CAD systems encompasses a wide spectrum of mammographic abnormalities, ranging from dense spiculated masses and architectural distortion to ill-defined masses and focal asymmetric densities. In some reports, breast cancers in which the predominant mammographic finding was a mass, asymmetric density, or architectural distortion were all combined into a single "mass" category for the purpose of study analysis (14,22). In these studies, CAD sensitivity values for mass detection ranged from 67% to 75%. Using more advanced CAD algorithms, Castellino et al (28) and Petrick et al (29) achieved mass detection sensitivity values of 86% and 87%, respectively. By comparison, we found that the sensitivity of CAD was 65% (47 of 72) for malignant masses and 80% (eight of 10) for malignant architectural distortions. The 80% accuracy rate for CAD detection of architectural distortion is similar to the 83% and 85% CAD accuracy rates reported in two previous studies (15,27). In contrast, Baker et al (30) found that fewer than half of 45 malignant lesions characterized by architectural distortion were correctly marked by either one of two commercially available CAD systems.
The majority of CAD-generated marks are determined to be associated with benign findings. In particular, false-positive CAD marks are frequently seen with vascular calcifications and pseudolesions caused by overlapping fibroglandular tissue. For the CAD system used in this study, the average number of false-positive marks has been reported to be 0.50.7 per image (22,30). Although most experienced radiologists can readily recognize and easily ignore these false-positive marks, the low specificity of the CAD system has potential negative effects on interpretations of screening mammograms. Most notably, these include an increase in the number of patients recalled for additional imaging and an increase in the number of breast biopsies with benign results (a decrease in the PPV for biopsy). In our study, CAD image review resulted in an absolute increase in the recall rate of 0.93%, from 9.84% to 10.77%. Although the statistical significance of this change was not determined in our study, the increase was considered an acceptable trade-off for the increase in breast cancer detection that occurred. The desirable goal for the recall rate at screening mammography, as currently suggested by the American College of Radiology, is less than 10% (24). The use of CAD to supplement screening mammographic interpretations resulted in 21 additional patient biopsies. The ratio of benign to malignant histologic diagnoses, however, was very similar to that for the biopsies performed on the basis of pre-CAD interpretations, and the PPV was minimally affected (41.0% before CAD vs 40.8% after CAD).
Our study was limited by the potential for observational bias that may have been introduced because interpreting radiologists were aware of their participation in the study. Readers' vigilance for detection of microcalcifications may have been reduced because of their awareness that the reported sensitivity of CAD for detection of grouped microcalcifications is quite high. Reliance on the CAD system to mark calcifications may have introduced a bias against the results without CAD. Because we did not record CAD-marked regions of interest that were not considered actionable by the interpreting radiologist, we could not determine whether the CAD system correctly marked findings of breast cancers that were dismissed by the radiologist as benign.
In conclusion, we found that the use of CAD analysis in the interpretation of 21 349 screening mammograms resulted in a 7.62% increase in the breast cancer detection rate. This was accomplished with an absolute increase in the recall rate of 0.93% (relative increase, 9.47%) and with essentially no effect on the PPV for biopsy (41.0% before CAD and 40.8% after CAD). The improvement in detection sensitivity was principally attributable to the presence of CAD marks on clusters of malignant calcifications that were not initially perceived by the radiologist. The CAD system marked 76.1% of the total breast cancers, and the radiologists detected 92.9%. The lower sensitivity of CAD in marking malignant masses and asymmetric densities underscores the fact that the absence of a CAD mark on any potential mammographic finding should not dissuade the radiologist from recalling a patient for additional imaging.
| ADVANCES IN KNOWLEDGE |
|---|
|
|
|---|
| ACKNOWLEDGMENTS |
|---|
| FOOTNOTES |
|---|
Abbreviations: CAD = computer-aided detection PPV = positive predictive value
Authors stated no financial relationship to disclose.
Author contributions: Guarantor of integrity of entire study, M.J.M.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, M.J.M., D.H.W.; clinical studies, all authors; statistical analysis, M.J.M., D.H.W., K.R.B.; and manuscript editing, all authors
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
R. M. Nishikawa and L. L. Pesce Computer-aided Detection Evaluation Methods Are Not Created Equal Radiology, June 1, 2009; 251(3): 634 - 636. [Full Text] [PDF] |
||||
![]() |
E. S. Burnside, J. Davis, J. Chhatwal, O. Alagoz, M. J. Lindstrom, B. M. Geller, B. Littenberg, K. A. Shaffer, C. E. Kahn Jr, and C. D. Page Probabilistic Computer Model Developed from Clinical Data in National Mammography Database Format to Classify Mammographic Findings Radiology, June 1, 2009; 251(3): 663 - 672. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. J. Gilbert, S. M. Astley, M. G.C. Gillan, O. F. Agbaje, M. G. Wallis, J. James, C. R.M. Boggis, S. W. Duffy, and the CADET II Group Single Reading with Computer-Aided Detection for Screening Mammography N. Engl. J. Med., October 16, 2008; 359(16): 1675 - 1684. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Gromet Comparison of Computer-Aided Detection to Double Reading of Screening Mammograms: Review of 231,221 Mammograms Am. J. Roentgenol., April 1, 2008; 190(4): 854 - 859. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Georgian-Smith, R. H. Moore, E. Halpern, E. D. Yeh, E. A. Rafferty, H. A. D'Alessandro, M. Staffa, D. A. Hall, K. A. McCarthy, and D. B. Kopans Blinded Comparison of Computer-Aided Detection with Human Second Reading in Screening Mammography Am. J. Roentgenol., November 1, 2007; 189(5): 1135 - 1141. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. F. Brem Blinded Comparison of Computer-Aided Detection with Human Second Reading in Screening Mammography: The Importance of the Question and the Critical Numbers Game Am. J. Roentgenol., November 1, 2007; 189(5): 1142 - 1144. [Full Text] [PDF] |
||||
![]() |
R. L. Ellis, A. A. Meade, M. A. Mathiason, K. M. Willison, and W. Logan-Young Evaluation of Computer-aided Detection Systems in the Detection of Small Invasive Breast Carcinoma Radiology, October 1, 2007; 245(1): 88 - 94. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Pruthi, K. R. Brandt, A. C. Degnim, M. P. Goetz, E. A. Perez, C. A. Reynolds, P. J. Schomberg, G. K. Dy, and J. N. Ingle A Multidisciplinary Approach to the Management of Breast Cancer, Part 1: Prevention and Diagnosis Mayo Clin. Proc., August 1, 2007; 82(8): 999 - 1012. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Ciatto, N. Houssami, D. Gur, R. M. Nishikawa, R. A. Schmidt, C. E. Metz, J. F. Ruiz, S. A. Feig, R. L. Birdwell, M. N. Linver, et al. Computer-aided screening mammography. N. Engl. J. Med., July 5, 2007; 357(1): 83 - 84. [Full Text] [PDF] |
||||
![]() |
L. Berlin Accuracy of Diagnostic Procedures: Has It Improved Over the Past Five Decades? Am. J. Roentgenol., May 1, 2007; 188(5): 1173 - 1178. [Full Text] [PDF] |
||||
![]() |
J. J. Fenton, S. H. Taplin, P. A. Carney, L. Abraham, E. A. Sickles, C. D'Orsi, E. A. Berns, G. Cutter, R. E. Hendrick, W. E. Barlow, et al. Influence of Computer-Aided Detection on Performance of Screening Mammography N. Engl. J. Med., April 5, 2007; 356(14): 1399 - 1409. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |