Radiology
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

DOI: 10.1148/radiol.2401050208
This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Horsch, K.
Right arrow Articles by Hendrick, R. E.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Horsch, K.
Right arrow Articles by Hendrick, R. E.
(Radiology 2006;240:357-368.)
© RSNA, 2006

Breast Imaging

Classification of Breast Lesions with Multimodality Computer-aided Diagnosis: Observer Study Results on an Independent Clinical Data Set1

Karla Horsch, PhD, Maryellen L. Giger, PhD, Carl J. Vyborny, MD, PhD{dagger}, Li Lan, MS, Ellen B. Mendelson, MD and R. Edward Hendrick, PhD

1 From the Department of Radiology, University of Chicago, 5841 S Maryland Ave, Chicago, IL 60637. From the 2002 RSNA Annual Meeting. Received February 23, 2005; revision requested April 21; revision received June 22; accepted July 18; final version accepted September 1. Supported in part by grants from the U.S. Army Medical Research and Material Command (DAMD 17-98-18194) and USPHS grants CA89452, RR11459, and T32 CA09649. Address correspondence to M.L.G. (e-mail: m-giger{at}uchicago.edu).


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 References
 
Purpose: To evaluate a computer-aided diagnosis multimodality intelligent workstation as an aid to radiologists in the interpretation of mammograms and breast sonograms.

Materials and Methods: An institutional review board approved the protocol for an observer study with signed consent, as well as the retrospective use of the mammograms, sonograms, and clinical data with waiver of consent. The HIPAA-compliant observer study was conducted with five breast radiologists and five breast imaging fellows, all of whom gave confidence ratings and patient management decisions, both without and with the computer aid, for 97 lesions that were unknown to both the observers and the computer. The performance of each observer without and with the computer aid was quantified by using four performance measures: area under the receiver operating characteristic curve (Az) value, partial Az value, sensitivity, and specificity. The statistical significance of the differences in the performance measures without and with the computer aid was determined by using a two-tailed t test for paired data.

Results: Use of the computer aid resulted in an improvement of the average performance of the 10 observers, as measured by means of a statistically significant increase in Az value (0.87–0.92; P < .001), partial Az value (0.47–0.68; P < .001), and sensitivity (0.88–0.93; P = .005). A statistically significant difference was not found in the specificity without and with the computer aid (0.66–0.69; P = .20).

Conclusion: Use of multimodality intelligent workstations can improve the performance of radiologists in the task of differentiating malignant and benign lesions at mammography and sonography.

© RSNA, 2006


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 References
 
Breast cancer is a leading cause of death among women in the United States (1). Early detection of breast cancer through mammographic screening of asymptomatic women has been shown to reduce mortality. However, the use of mammography for diagnosis is associated with a large number of unnecessary biopsy procedures, which results in higher costs and greater patient discomfort (2,3).

Breast sonography is used as an important adjunct to diagnostic mammography and is typically performed to evaluate palpable or mammographically identified masses to determine whether they are cystic or solid. The reported accuracy of the diagnosis of simple cysts at sonography is 96%–100% (4). Because of the large overlap in the appearance of malignant and benign solid lesions at sonography, however, differentiation among solid masses may be difficult (5,6).

Various investigators are developing computerized image analysis methods for the characterization and diagnosis of lesions on breast images (7–13). The ultimate evaluation of such computer outputs, however, is in the assessment of their additive value when used as an aid by radiologists. Results of previous observer studies have demonstrated the benefit of such mammographic (14–16) and sonographic (17) automatic classification schemes as aids to radiologists in differentiating malignant and benign breast disease. In each of these studies, the performances of the radiologists, in terms of area under the receiver operating characteristic (ROC) curve (Az), showed a statistically significant improvement when the computer aid was used.

Independent double readings by two radiologists have also been investigated as a means of improving the performance of radiologists in both detection and diagnosis of breast cancer (18,19). When the two radiologists disagree about patient management, various actions can occur, such as consultation with a third reader who acts as the "tiebreaker." In screening mammography, independent double readings have been shown to improve sensitivity in the detection of breast cancer (7). In diagnostic mammography, Jiang et al (19) have compared radiologists' (a) single-reading performance without computer aid, (b) single-reading performance with computer aid, and (c) simulated unaided double-reading performance in the task of differentiating malignant and benign microcalcifications. They found that the use of a computer aid improved the diagnostic accuracy of radiologists more than did the simulated independent double readings.

The purpose of our study was to evaluate the use of a computer-aided diagnosis multimodality intelligent workstation as an aid to radiologists in the interpretation of mammograms and breast sonograms.


   MATERIALS AND METHODS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
RESULTS
 DISCUSSION
 APPENDIX
 References
 
M.L.G. and C.J.V. are shareholders in R2 Technology (Los Altos, Calif). It is the University of Chicago Conflict of Interest Policy that investigators disclose publicly actual or potential significant financial interest that would reasonably appear to be directly and significantly affected by the research activities.

Databases
Three databases of breast lesions were used in this study, as follows: a database of mammographic images, a database of sonographic images, and an independent multimodality database in which each lesion was represented by both mammograms and sonograms of the breast. An institutional review board approved the protocol for the retrospective use of the mammograms, sonograms, and clinical data, as well as the protocol. Informed consent was waived for use of these studies and data. The institutional review board approved the observer study, with signed informed consent from the observers. Health Insurance Portability and Accountability Act, or HIPAA, compliance was observed on implementation of HIPAA. The mammographic and sonographic databases were used to train our mammographic and sonographic computer-aided diagnosis classifiers, respectively. The mammographic and sonographic databases were also used as the mammographic and sonographic reference libraries (online atlases) for our intelligent workstation. The independent multimodality database was used for the observer study. The number and type of lesions in each of the three databases are given in Table 1.


View this table:
[in this window]
[in a new window]

  		Table 1. Number of Cases and Types of Lesions in the Three Databases

 
Our mammographic training database is a combination of previously collected databases; the images were retrospectively collected from the University of Chicago Hospitals, and all lesions had undergone biopsy (16,20,21).

The source for our sonographic and multimodality databases is a set of 455 lesions represented by 909 consecutive sonographic images. These images were retrospectively collected from the Lynn Sage Comprehensive Breast Center of Northwestern Memorial Hospital (Chicago, Ill) and represent biopsy- or aspiration-proved lesions. The images were obtained with a broadband linear-array transducer (10–5 MHz) (ATL 3000; Philips, Bothell, Wash) and were captured directly from the 8-bit video signal. The number of images per lesion varied from one to six. An experienced breast radiologist (C.J.V., with 18 years of clinical experience) reviewed the patient folders corresponding to the 455 sonographic lesions and selected mammograms on which a mammographically evident lesion corresponded to a lesion in the sonographic database. In addition, for each lesion used in the observer study, we required at least one standard view with a mammographically evident lesion. These mammograms were then digitized at 0.1 mm/pixel by using a Lumisys film digitizer (Lumisys, Sunnyvale, Calif). The multimodality database consists of 105 lesions represented by both mammograms and sonograms of the breast. Of these 105 lesions in the multimodality database, 18 were used to train the observers in the use of the workstation. Our multimodality testing database consisted of the remaining 97 lesions (Table 1).

Multimodality Intelligent Workstation
Our intelligent workstation displays four-on-one whole-breast mammograms, with the regions of interest containing a given lesion from the standard and special mammographic views, and the sonographic views. The mammographic and sonographic computer output is presented in three different ways (22). First, for a given lesion, separate estimates of the probability of malignancy (PM) calculated from that lesion's mammographic images and that lesion's sonographic images are given numerically. Second, for a given image of a given lesion, the computer analysis is presented pictorially through a color-coded display of images automatically selected from either the mammographic or the sonographic reference libraries. The similarity index used in this automatic search and retrieval is based on either the probabilities of malignancy or the feature values calculated from the given image. The user is allowed to interactively choose between the available mammographic and sonographic images, as well as between similarity indexes (feature values and PM). Malignant and benign reference library images are outlined in red and green, respectively. Third, for a given lesion, the computer analysis is also presented graphically. That lesion's mammographic or sonographic PM or feature value is shown relative to the histogram of malignant and benign images from the appropriate reference library (Fig 1).


Figure 1
View larger version (93K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1a: The intelligent workstation interface used in our observer study for a malignant lesion with a mammographic and sonographic computer PM estimate of 91.67% and 76.07%, respectively. Shown are the (a) mammographic and (b) sonographic computer outputs.

 

Figure 1
View larger version (82K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1b: The intelligent workstation interface used in our observer study for a malignant lesion with a mammographic and sonographic computer PM estimate of 91.67% and 76.07%, respectively. Shown are the (a) mammographic and (b) sonographic computer outputs.

 
The automatic mammographic and sonographic computer analyses are based on the output of our mammographic and sonographic computer classifiers, respectively, both of which have been described in detail elsewhere (mammographic classifier [16,23,24], sonographic classifier [17,25–27]). By using a manually indicated lesion center, the computer automatically segments the lesion from the normal tissue and automatically extracts lesion features. The mammographic classifier extracts features quantifying spiculation, margin sharpness, texture, shape, and gray level in some computer-defined neighborhood of the lesion, while the sonographic classifier extracts features quantifying lesion shape, margin, texture (based on autocorrelation of the lesion's gray-level values), and posterior acoustic behavior. Bayesian neural networks with three and four hidden nodes are then used to merge the five mammographic and the four sonographic computer-extracted features, respectively. The two computer classifiers were used to determine the computer's PM from the mammograms and sonograms for the 97 lesions in the observer study database. Recall that these 97 lesions are independent of both the mammographic and sonographic training databases.

The Az value of each classifier on its training database, as well as on the multimodality observer study database, was determined by using LABROC4 (C. Metz, University of Chicago, Ill) (28). The Az values of the mammographic computer classifier on the mammographic training database and on the multimodality observer study database are 0.89 (resubstitution) and 0.81 (independent testing), respectively. The Az values of the sonographic computer classifier on the sonographic training database and on the multimodality observer study database are 0.89 (resubstitution) and 0.93 (independent testing), respectively.

Observer Study
Five breast radiologists (12, 15, 16, 31, and 2 years of clinical experience) and five breast imaging fellows participated in the observer study. The radiologist who helped to collect the multimodality database by reviewing patient folders was not an observer. The fellows were at the end of a 1-year breast imaging training program. Each observer completed a single interpretation session. Before beginning the study, all observers (a) signed an institutional review board–approved consent form, (b) were given a brief description of mammographic and sonographic features used by the computer classifiers, (c) were told that the mammographic and sonographic PM estimates reflected a prevalence of around 50%, and (d) were given the information in Table 2, which shows the mammographic and sonographic computer performance in terms of sensitivity and specificity at various cutoff levels. Observers were allowed to ask questions concerning the interpretation of the performance table. It is important to note that to effectively interpret the information given by a diagnostic tool, a person needs to know how well that tool performs the diagnostic task. It is also important to remember, when considering the specificities in the performance table, that the computer performance is for databases consisting only of biopsy-proved lesions.


View this table:
[in this window]
[in a new window]

  		Table 2. Computer Performance Table Given to the Observers

 
During the study's interpretation session, for each lesion, each observer was sequentially exposed to two reading conditions: mammograms and sonograms without the intelligent workstation output (unaided) and mammograms and sonograms with the intelligent workstation output (aided). For each condition, each radiologist was asked (a) "What is your confidence that the lesion is malignant on a continuous scale from 0% to 100%?" and (b) "What is your recommendation for patient management—ie, follow-up or biopsy?" After assessing the lesion without aid, the radiologists were asked to consider the intelligent workstation output and modify their confidence rating and/or patient management recommendation accordingly. This system of evaluation, considering the lesion first without aid and then with aid, mimics the manner in which radiologists are expected to use the computer aid clinically (29).

Before the observers were asked to evaluate the 97 lesions from the testing database, they evaluated 18 lesions from the training database (recall that these are independent of the 97 lesions for the testing database), which consisted of four complicated cysts, seven benign solid lesions, and seven carcinomas. For these 18 lesions only, the observers were given feedback on the pathologic diagnosis after completing the evaluation of each lesion—that is, after both the unaided and aided reading conditions. The order of display for the images representing the "training lesions" was the same for all observers, while the order of display for the images representing the 97 observer study lesions was randomized. During the observer study, each observer was allowed to interactively change the contrast and brightness of the display according to his or her preference. No time limit was imposed.

Statistical Analysis
To quantify the performance of each observer in the task of distinguishing malignant from benign lesions, both unaided and aided, four performance measures were computed for the unaided and aided reading conditions. ROC analysis (30) was used to analyze the ratings representing the radiologist's confidence of whether a lesion is malignant. The Az value and the partial area index at 0.90 sensitivity (hereafter called the partial Az value) (31) were chosen as performance measures. In addition, the performance measures of sensitivity and specificity were determined for each observer on the basis of the patient management decision of whether to perform biopsy. The average performance measures were also determined for the three groups of observers: the five breast radiologists, the five fellows, and all 10 observers. The differences in the performance measures obtained from the unaided and aided observer data quantify the additive effect of the computer aid on the observers' performance in recommending biopsy for malignant lesions and in differentiating malignant lesions from benign lesions. A two-tailed t test for paired data (Excel 2004; Microsoft, Redmond, Wash) (32) was used to determine the significance (P < .05) of these differences in the average performance measures without aid and with the intelligent workstation aid for the three groups of observers.

To quantify the changes in patient management decision with the intelligent workstation aid, we computed for each radiologist the number of malignant, benign solid, and cystic lesions for which the patient management decision was changed with the addition of computer aid. These measurements of change in patient management decision were then averaged over the three groups of observers.

To analyze the relation between the computer's PM estimates and a change in the patient management decision, we estimated for both malignant and benign lesions the conditional probability of a particular change in patient management decision assuming (conditioned on) a correct computer classification given a particular cutoff value. Recall that to use either the mammographic or the sonographic PM estimates to classify malignant and benign lesions, a cutoff value must be set and all lesions with a PM greater than the cutoff value are classified as malignant and all other lesions are classified as benign. We considered three types of correct classification, as follows: (a) correct mammographic computer classification, independent of the sonographic computer classification, (b) correct sonographic computer classification, independent of the mammographic computer classification, and (c) correct mammographic and sonographic computer classification. The Appendix gives details on the estimation of these conditional probabilities. Unfortunately, we are not aware of a method for determining the error on the estimated conditional probabilities (see the Appendix for comments). These estimates of conditional probability are displayed graphically.

Because of the importance of not missing a cancer, the malignant lesions for which at least one observer changed his or her patient management decision with the addition of computer aid are of special interest. We want to understand how the computer classification of malignant and benign lesions relates to the observers' patient management decisions for these malignant lesions. In particular, since most clinics would like to function at high sensitivity during diagnostic work-up, it is interesting to compare the mammographic (or sonographic) computer PM estimates to the cutoff value that produces a mammographic (or sonographic) computer classifier with a 98% sensitivity on the training database. The mammographic cutoff (denoted as 0.98CM) is 14% and the sonographic cutoff (denoted as 0.98CS) is 16%, since these cutoffs produce mammographic and sonographic classifiers both with a sensitivity of 98% (and specificities of 28% and 56%, respectively) on their training databases. The cutoffs 0.98CM and 0.98CS produced mammographic and sonographic classifiers with sensitivities of 92% and 100% (and specificities of 31% and 59%), respectively, on the multimodality observer study database.


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
DISCUSSION
 APPENDIX
 References
 
Performance of Observers
The ROC curves for each of the three groups in the task of distinguishing malignant from benign lesions (Fig 2), without computer aid and with aid, are determined by averaging the a and b values obtained from LABROC4 (28) for each observer.


Figure 2
View larger version (41K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2a: ROC curves for (a) breast radiologists, (b) fellows, and (c) all observers.

 

Figure 2
View larger version (40K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2b: ROC curves for (a) breast radiologists, (b) fellows, and (c) all observers.

 

Figure 2
View larger version (40K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2c: ROC curves for (a) breast radiologists, (b) fellows, and (c) all observers.

 
Use of the intelligent workstation aid improved the performance (Table 3) of all three groups of observers in terms of the Az value at statistically significant levels (P < .05) in the task of distinguishing malignant and benign lesions (breast radiologists, Az of 0.87–0.91; fellows, 0.88–0.93; all observers, 0.87–0.92). The partial Az value for all three groups also improved at statistically significant levels with the intelligent workstation aid (breast radiologists, partial Az of 0.49–0.64; fellows, 0.45–0.71; all observers, 0.47–0.68).


View this table:
[in this window]
[in a new window]

  		Table 3. Performance of Individual Observers and Average Performance of the Groups of Observers without Computer Aid and with Aid

 
Although the sensitivity and specificity of all three groups of observers improved with aid, only the improvements in sensitivity of the breast imaging fellows and of all observers were statistically significant (fellows, sensitivity of 0.83–0.90; all observers, 0.88–0.93).

Patient Management
Three examples of patient management changes are shown in Figures 3–5. On average, all observers changed the patient management decision for 2.5 malignant lesions from follow-up to biopsy and for 0.6 malignant lesions from biopsy to follow-up (Table 4). On the other hand, on average, all observers changed the patient management decision for 2.0 benign solid lesions from follow-up to biopsy and for 3.3 benign solid lesions from and biopsy to follow-up.


Figure 3
View larger version (138K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3a: Images of a malignant lesion for which two breast radiologists and three fellows changed the management decision from follow-up to biopsy after computer aid. Mammographic and sonographic computer PM estimates for this case were 87.85% and 70.60%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 3
View larger version (164K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3b: Images of a malignant lesion for which two breast radiologists and three fellows changed the management decision from follow-up to biopsy after computer aid. Mammographic and sonographic computer PM estimates for this case were 87.85% and 70.60%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 3
View larger version (169K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3c: Images of a malignant lesion for which two breast radiologists and three fellows changed the management decision from follow-up to biopsy after computer aid. Mammographic and sonographic computer PM estimates for this case were 87.85% and 70.60%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 3
View larger version (168K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3d: Images of a malignant lesion for which two breast radiologists and three fellows changed the management decision from follow-up to biopsy after computer aid. Mammographic and sonographic computer PM estimates for this case were 87.85% and 70.60%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 4
View larger version (164K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 4a: Images of a malignant lesion for which one breast radiologist and two fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 6.85% and 24.80%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a transverse sonographic view, and (c) a longitudinal sonographic view. The craniocaudal mammographic view was unavailable.

 

Figure 4
View larger version (168K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 4b: Images of a malignant lesion for which one breast radiologist and two fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 6.85% and 24.80%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a transverse sonographic view, and (c) a longitudinal sonographic view. The craniocaudal mammographic view was unavailable.

 

Figure 4
View larger version (167K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 4c: Images of a malignant lesion for which one breast radiologist and two fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 6.85% and 24.80%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a transverse sonographic view, and (c) a longitudinal sonographic view. The craniocaudal mammographic view was unavailable.

 

Figure 5
View larger version (165K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 5a: Images of a benign lesion for which one breast radiologist and three fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 8.80% and 7.09%, respectively. Shown are (a) a region of interest from the mediolateral oblique mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 5
View larger version (138K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 5b: Images of a benign lesion for which one breast radiologist and three fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 8.80% and 7.09%, respectively. Shown are (a) a region of interest from the mediolateral oblique mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 5
View larger version (154K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 5c: Images of a benign lesion for which one breast radiologist and three fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 8.80% and 7.09%, respectively. Shown are (a) a region of interest from the mediolateral oblique mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

Figure 5
View larger version (164K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 5d: Images of a benign lesion for which one breast radiologist and three fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 8.80% and 7.09%, respectively. Shown are (a) a region of interest from the mediolateral oblique mammographic view, (b) a region of interest from the craniocaudal mammographic view, (c) a transverse sonographic view, and (d) a longitudinal sonographic view.

 

View this table:
[in this window]
[in a new window]

  		Table 4. Change in Patient Management Decision after Use of Computer Aid

 
For eight of the malignant lesions, at least one observer correctly changed the management decision from follow-up to biopsy after use of the computer aid (Table 5). These eight lesions are correctly classified by both the mammographic and sonographic computer classifiers resulting from the cutoffs 0.98CM and 0.98CS, respectively (see Materials and Methods). However, for four of the malignant lesions, at least one observer incorrectly changed from biopsy to follow-up after use of the computer aid. Both the mammographic and sonographic classifiers resulting from the cutoffs 0.98CM and 0.98CS, respectively, correctly classified three of these four lesions as malignant. Only one malignant lesion at mammography was incorrectly classified by the computer as benign (the PM estimate of 7% for malignant case 12 was less than the cutoff 0.98CM of 14%). This malignant lesion was also the case for which the most observers incorrectly changed the patient management decision from biopsy to follow-up after use of the computer aid.


View this table:
[in this window]
[in a new window]

  		Table 5. Malignant Lesions for Which at Least One Observer Changed Patient Management Decision after Use of Computer Aid

 
Use of the computer aid resulted in at least one observer correctly changing the management decision from biopsy to follow-up for 24 of the benign lesions and incorrectly changing it from follow-up to biopsy for 14 of the benign lesions.

For both malignant and benign lesions, for those lesions correctly classified by the computer-given cutoffs between 5% and 30%, the estimated conditional probability of making a correct change in patient management decision is greater than that for making an incorrect change (Fig 6). For malignant lesions, the estimated conditional probability curves, conditioned on any of the three types of correct classification, were quite similar for cutoffs levels between 5% and 30%. However, for benign lesions, the type of correct classification matters. For cutoffs between 5% and 30%, observers were more likely to make a correct change in patient management decision for those benign lesions correctly classified by both computer classifiers than to make a correct change in patient management decision for those benign lesions correctly classified by either the mammographic or the sonographic computer classifier (independent of the other classifier).


Figure 6
View larger version (20K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 6a: Graphs of conditional probability between biopsy (BX) and follow-up (FU). (a, b) Probability of changing from (a) follow-up to biopsy and (b) biopsy to follow-up for malignant cases that the computer correctly classifies. (c, d) Probability of changing from (c) follow-up to biopsy and (d) biopsy to follow-up on benign cases that the computer correctly classifies. Thick black line = correct mammographic and sonographic classification; thin black line = correct mammographic computer classification; gray line = correct sonographic classification, given cutoff.

 

Figure 6
View larger version (19K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 6b: Graphs of conditional probability between biopsy (BX) and follow-up (FU). (a, b) Probability of changing from (a) follow-up to biopsy and (b) biopsy to follow-up for malignant cases that the computer correctly classifies. (c, d) Probability of changing from (c) follow-up to biopsy and (d) biopsy to follow-up on benign cases that the computer correctly classifies. Thick black line = correct mammographic and sonographic classification; thin black line = correct mammographic computer classification; gray line = correct sonographic classification, given cutoff.

 

Figure 6
View larger version (21K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 6c: Graphs of conditional probability between biopsy (BX) and follow-up (FU). (a, b) Probability of changing from (a) follow-up to biopsy and (b) biopsy to follow-up for malignant cases that the computer correctly classifies. (c, d) Probability of changing from (c) follow-up to biopsy and (d) biopsy to follow-up on benign cases that the computer correctly classifies. Thick black line = correct mammographic and sonographic classification; thin black line = correct mammographic computer classification; gray line = correct sonographic classification, given cutoff.

 

Figure 6
View larger version (27K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 6d: Graphs of conditional probability between biopsy (BX) and follow-up (FU). (a, b) Probability of changing from (a) follow-up to biopsy and (b) biopsy to follow-up for malignant cases that the computer correctly classifies. (c, d) Probability of changing from (c) follow-up to biopsy and (d) biopsy to follow-up on benign cases that the computer correctly classifies. Thick black line = correct mammographic and sonographic classification; thin black line = correct mammographic computer classification; gray line = correct sonographic classification, given cutoff.

 

   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
APPENDIX
 References
 
When we consider the computer or human performance in the task of distinguishing malignant from benign lesions in any of the three databases, it is important to remember that all three databases consist only of biopsy-proved or aspiration-proved cases. This means that the specificities of the radiologists (ie, the radiologists who actually performed the clinical work-up, as opposed to those who performed the observer study) on these databases were zero, as evidenced by the recommendation for biopsy or aspiration in all cases in these databases.

In terms of Az value, the breast radiologists and fellows demonstrated similar performances without the computer aid and also showed similar improvements with the computer aid. However, the average operating points of the breast radiologists and fellows for both reading conditions are quite different, with the breast radiologists operating, on average, at higher sensitivity and lower specificity than the fellows (breast radiologists without aid, sensitivity of 0.93 and specificity of 0.57; fellows without aid, sensitivity of 0.83 and specificity of 0.75; breast radiologists with aid, sensitivity of 0.96 and specificity of 0.58; fellows with aid, sensitivity of 0.90 and specificity of 0.80).

On average, each of the three groups changed more of the patient management for cases of actually benign solid and actually cystic lesions from biopsy to follow-up than from follow-up to biopsy after use of computer aid. Likewise, on average, each of the three groups changed more of the patient management for cases of actually malignant lesions from follow-up to biopsy than from biopsy to follow-up after use of computer aid. This is a promising finding that supports the adoption of the workstation in clinical use.

There are two primary limitations to this observer study. First, we did not include in our observer study any lesions that had not undergone biopsy or aspiration in their clinical evaluation. This is a limitation because the ultimate goal of this research is clinical use of the computer workstation for the evaluation and patient management decision for detected lesions—those that will subsequently undergo biopsy or aspiration, as well as those that will not. The inclusion of lesions that had not undergone biopsy or aspiration in an observer study might, for example, affect the sensitivity and specificity of the radiologist user.

The second limitation is that the study findings cannot be used to determine whether the radiologists' performances with the multimodality workstation aid are significantly improved in comparison with their performances with the mammographic (single-modality) computer aid system. Another observer study is necessary, in which three reading conditions would be evaluated: reading without computer aid, reading with just mammographic computer aid, and reading with mammographic and sonographic multimodality computer aid. We note that we previously demonstrated in an observer study that use of computer-aided diagnosis with mammography improves the performance of radiologists in the task of differentiating malignant and benign lesions (16). Although the mammographic computer classifier is the same in the previous and the current study, the observer study databases are different. Therefore, the previous and current study cannot be compared.

Currently, our multimodality breast computer-aided diagnosis workstation runs online for the analysis of mammographic and sonographic images.

We have shown that the use of our multimodality intelligent workstation improved the performance of both breast radiologists and breast imaging fellows in the task of differentiating malignant from benign breast lesions on mammograms and sonograms. Both breast imaging radiologists and breast imaging fellows showed statistically significant improvement in Az and partial Az values. In addition, the sensitivity of the breast imaging fellows also improved at a statistically significant level. Such multimodality intelligent workstations, therefore, hold promise for the improvement of the performance of radiologists in the task of differentiating malignant and benign breast lesions on mammograms and sonograms, and transition and further validation of multimodality workstations in the clinical arena are warranted.


   APPENDIX
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 APPENDIX
References
 
We want to estimate, for both malignant and benign cases, the conditional probability of a particular change in patient management decision assuming (conditioned on) a correct computer classification given a particular cutoff value. By using the patient management decisions from observers, where N indicates the number of observers, we estimate the conditional probability of making patient management decision d1 without aid and patient management decision d2 with aid, assuming (conditioned on) cases in some set, S:

Formula
where Y is the number of cases in S and X is the number of cases in S for which observer i makes decision d1 without aid and decision d2 with aid. The conditional probabilities of interest satisfy either d1 = follow-up and d2 = biopsy or d1 = biopsy and d2 = follow-up as shown in Figure A1.


Figure 1
View larger version (60K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure A1: Sets used to estimate conditional probability of a change in patient management, conditioned on a correct computer classification given a particular cutoff. PMM = PM at mammography, PMS = PM at sonography.

 
All together, there are 12 conditional probabilities. We note that if the elements in the sum given in the equation above were independent, we could compute the error on p(d1,d2|S) from the theory of binomial distributions. Unfortunately, the elements are not independent. We are not aware of a method to compute the error in the case of dependence.


Figure 4
View larger version (168K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 4d: Images of a malignant lesion for which one breast radiologist and two fellows changed the management decision from biopsy to follow-up after computer aid. Mammographic and sonographic computer PM estimates for this case were 6.85% and 24.80%, respectively. Shown are (a) a region of interest from the mediolateral mammographic view, (b) a transverse sonographic view, and (c) a longitudinal sonographic view. The craniocaudal mammographic view was unavailable.

 

   ACKNOWLEDGMENTS
 
We thank Charles E. Metz, PhD, for discussion useful in understanding the conditional probability of a change in patient management decision used in the discussion section. We also thank all of the radiologists and fellows for their participation and comments.


   FOOTNOTES
 

Abbreviations: Az = area under the ROC curve • PM = probability of malignancy • ROC = receiver operating characteristic

{dagger} Deceased. Back

See Materials and Methods for pertinent disclosures.

Author contributions: Guarantors of integrity of entire study, K.H., M.L.G.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, K.H., M.L.G.; experimental studies, K.H., M.L.G., C.J.V., L.L.; statistical analysis, K.H., M.L.G.; and manuscript editing, K.H., M.L.G., L.L., E.B.M., R.E.H.


   References
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 References
 

  1. Jemal A, Tiwari RC, Murray T, et al. Cancer statistics, 2004. CA Cancer J Clin 2004;54:8–29.[Abstract/Free Full Text]
  2. Dennis MA, Parker SH, Klaus AJ, Stavros AT, Kaske TI, Clark SB. Breast biopsy avoidance: the value of normal mammograms and normal sonograms in the setting of a palpable lump. Radiology 2001;219:186–191.[Abstract/Free Full Text]
  3. Burnside ES, Sickles EA, Sohlich RE, Dee KE. Differential value of comparison with previous examinations in diagnostic versus screening mammography. AJR Am J Roentgenol 2002;179:1173–1177.[Abstract/Free Full Text]
  4. Jackson VP. The role of ultrasound in breast imaging. Radiology 1990;177:305–311.[Free Full Text]
  5. Stavros AT, Thickman D, Rapp CL, Dennis MA, Parker SH, Sisney GA. Solid breast nodules: use of sonography to distinguish between benign and malignant lesions. Radiology 1995;196:123–134.[Abstract/Free Full Text]
  6. American College of Radiology. Breast imaging reporting and data system (BI-RADS): ultrasound. Reston, Va: American College of Radiology, 2003.
  7. Giger ML. Computer-aided diagnosis. In: Haus AG, Yaffe MJ, eds. Syllabus: technical aspects of mammography. Oak Brook, Ill: Radiological Society of North America, 1993; 283–298.
  8. Giger ML, Huo Z, Kupinski MA, Vyborny CJ. Computer-aided diagnosis in mammography. In: Fitzpatrick JM, Sonka M, eds. Handbook of medical imaging. Vol 2. Bellingham, Wash: SPIE Press, 2000; 915–1004.
  9. Vyborny CJ, Giger ML. Computer vision and artificial intelligence in mammography. AJR Am J Roentgenol 1994;162:699–708.[Abstract/Free Full Text]
  10. Vyborny CJ, Giger ML, Nishikawa RM. Computer-aided detection and diagnosis of breast cancer. Radiol Clin North Am 2000;38:725–740.[CrossRef][Medline]
  11. Jiang Y. Classification of breast lesions in mammograms. In: Bankman, ed. Handbook of medical imaging, processing and analysis. San Diego, Calif: Academic Press, 2000; 341–358.
  12. Giger ML. Computerized image analysis in breast cancer detection and diagnosis. Semin Breast Dis 2002;5:199–210.
  13. Giger ML. CAD in diagnostic mammography and multimodality breast imaging. In: Karellas A, Giger ML, eds. RSNA categorical course in diagnostic radiology physics: physical aspects of breast imaging. Oak Brook, Ill: Radiological Society of North America, 2004; 205–217.
  14. Jiang Y, Nishikawa RM, Schmidt RA, Metz CE, Giger ML, Doi K. Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol 1999;6:22–33.[CrossRef][Medline]
  15. Chan HP, Sahiner B, Helvie MA, et al. Improvement of radiologist's characterization of mammographic masses by using computeraided diagnosis: an ROC study. Radiology 1999;212:817–827.[Abstract/Free Full Text]
  16. Huo Z, Giger ML, Vyborny CJ, Metz CE. Breast cancer: effectiveness of computer-aided diagnosis—observer study with independent database of mammograms. Radiology 2002;224:560–568.[Abstract/Free Full Text]
  17. Horsch K, Giger ML, Vyborny CJ, Venta LA. Performance of computer-aided diagnosis in the interpretation of lesions on breast sonography. Acad Radiol 2004;11:272–280.[CrossRef][Medline]
  18. Thurfjell EL, Lernevall KA, Taube AA. Benefit of independent double reading in a population-based mammography screening program. Radiology 1994;191:241–244.[Abstract/Free Full Text]
  19. Jiang Y, Nishikawa RM, Schmidt RA, Metz CE, Doi K. Relative gains in diagnostic accuracy between computer-aided diagnosis and independent double reading. In: Proceedings of SPIE: medical imaging 2000. Vol 3981. Bellingham, Wash: International Society for Optical Engineering, 2000; 10–15.
  20. Huo Z, Giger ML, Vyborny CJ, Wolverton DE, Metz CE. Computerized classification of benign and malignant masses on digitized mammographic masses: a study of robustness. Acad Radiol 2000;7:1077–1084.[CrossRef][Medline]
  21. Huo Z, Giger ML, Vyborny CJ. Computerized analysis of multiple-mammographic views: potential usefulness of special view mammograms in computer-aided diagnosis. IEEE Trans Med Imaging 2001;20:1285–1292.[CrossRef][Medline]
  22. Giger ML, Huo Z, Lan L, Vyborny CJ. Intelligent search workstation for computer aided diagnosis. In: Proceedings of Computer Assisted Radiology and Surgery (CARS) 2000. Philadelphia, Pa: Elsevier, 2000; 822–827.
  23. Huo Z, Giger ML, Vyborny CJ, et al. Analysis of spiculation in the computerized classification of mammographic masses. Med Phys 1995;22:1569–1579.[CrossRef][Medline]
  24. Huo Z, Giger ML, Vyborny CJ, Woverton DE, Schmidt RA, Doi K. Automated computerized classification of malignant and benign masses on digitized mammograms. Acad Radiol 1998;5:155–168.[CrossRef][Medline]
  25. Giger ML, Al-Hallaq H, Huo Z, et al. Computerized analysis of lesions in US images of the breast. Acad Radiol 1999;6:665–674.[CrossRef][Medline]
  26. Horsch K, Giger ML, Venta LA, Vyborny CJ. Automatic segmentation of breast lesions on ultrasound. Med Phys 2001;28(8):1652–1659.[CrossRef][Medline]
  27. Horsch K, Giger ML, Venta LA, Vyborny CJ. Computerized diagnosis of breast lesions on ultrasound. Med Phys 2002;29(2):157–164.[CrossRef][Medline]
  28. LABROC4. http://xray.bsd.uchicago.edu/krl/roc_soft.htm. Updated January 12, 2006.
  29. Metz C. Fundamental ROC analysis. In: Beutel J, Metter R, Kundel H, eds. Handbook of medical imaging. Vol 1. Bellingham, Wash: SPIE Press, 2000; 751–764.
  30. Metz CE. ROC methodology in radiologic imaging. Invest Radiol 1986;21:720–733.[Medline]
  31. Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristics partial area index for highly sensitive diagnostic tests. Radiology 1996;201:745–750.[Abstract/Free Full Text]
  32. Hays WL. Statistics. Philadelphia, Pa: Harcourt Brace College, 1999.



This article has been cited by other articles:


Home page
 radtechHome page
T. G. ODLE
Breast Ultrasound
Radiol. Technol., January 1, 2007; 78(3): 222M - 242M.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Horsch, K.
Right arrow Articles by Hendrick, R. E.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Horsch, K.
Right arrow Articles by Hendrick, R. E.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
RADIOLOGY RADIOGRAPHICS RSNA JOURNALS ONLINE