DOI: 10.1148/radiol.2422051997
(Radiology 2007;242:396-402.)
© RSNA, 2007
Holistic Component of Image Perception in Mammogram Interpretation: Gaze-tracking Study1
Harold L. Kundel, MD,
Calvin F. Nodine, PhD,
Emily F. Conant, MD and
Susan P. Weinstein, MD
1 From the Department of Radiology, University of Pennsylvania Health System, 3600 Market St, Suite 370, Philadelphia, PA 19104. Received December 12, 2005; revision requested January 24, 2006; revision received March 2; accepted April 11; final version accepted June 1.
Address correspondence to H.L.K. (e-mail: kundelh{at}uphs.upenn.edu).
 |
ABSTRACT
|
|---|
Purpose: To test the hypothesis that rapid and accurate performance of the proficient observer in mammogram interpretation involves a shift in the mechanism of image perception from a relatively slow search-to-find mode to a relatively fast holistic mode.
Materials and Methods: This HIPAA-compliant study had institutional review board approval, and participant informed consent was obtained; patient informed consent was not required. The eye positions of three full-time mammographers, one attending radiologist, two mammography fellows, and three radiology residents were recorded during the interpretation of 20 normal and 20 subtly abnormal mammograms. The search time required to first locate a cancer, as well as the initial eye scan path, was determined and compared with diagnostic performance as measured with receiver operating characteristic (ROC) analysis.
Results: The median time for all observers to fixate a cancer, regardless of the decision outcome, was 1.13 seconds, with a range of 0.68 second to 3.06 seconds. Even though most of the lesions were fixated, recognition of them as cancerous ranged from 85% (17 of 20) to 10% (two of 20), with corresponding areas under the ROC curve of 0.87–0.40. The ROC index of detectability, da, was linearly related to the time to first fixate a cancer with a correlation (r2) of 0.81.
Conclusion: The rapid initial fixation of a true abnormality is evidence for a global perceptual process capable of analyzing the visual input of the entire retinal image and pinpointing the spatial location of an abnormality. It appears to be more highly developed in the most proficient observers, replacing the less efficient initial search-to-find strategies.
© RSNA, 2007
|
INTRODUCTION
|
|---|
Visual recognition skill is an important component of expertise in radiology, but its development is not well understood. Experienced radiologists recognize abnormalities more rapidly and more accurately than do residents and fellows. The time course for making positive decisions on both chest radiographs and mammograms has been modeled by measuring the change in the positive predictive value with time and fitting the data to a biexponential model that shows a fast and a slow component (1,2).
We surmise that the more rapid performance of the most proficient radiologists involves a shift in the mechanism of image perception from a relatively slow search-to-find mode to a relatively fast holistic mode. The holistic mode, analogous to the mechanism used for the recognition of faces, depends on an initial global analysis of the retinal image that leads to the identification of image perturbations that are consistent with abnormality—basically, distinguishing between normal and abnormal (3,4). The gaze is then directed to the perturbations, and focal feature analysis either incorporates the perturbation into the perception of the image or disregards it (5).
Once the initially identified perturbations are resolved, the observer searches the image for other potential abnormalities that were not sufficiently conspicuous to be flagged in the initial global analysis. This starts the search-to-find mode that is accomplished by scanning the gaze over the image in jumps, usually called saccades, with pauses for a few 100 msec to fixate salient regions (6). Saccades are ballistic—that is, the landing spot for the new fixation is determined before the saccade is launched (7,8). Therefore, at the onset of scanning, the only visual information available to the eye-brain system about where to aim and land the first saccade must be derived from a global retinal analysis.
Once scanning is in progress, it is difficult to separate those perturbations that were identified initially and stored for later checking from those that are newly identified either globally or as a result of focal search. Global retinal analysis and focal feature analysis are simultaneously active during the fixations; the global analysis maintains the stability of the perception and discovers new perturbations that require analysis, while the focal feature analysis resolves the perturbations.
The purpose of our study was to test the hypothesis that rapid and accurate performance of the proficient observer involves a shift in the mechanism of image perception from a relatively slow search-to-find mode to a relatively fast holistic mode.
|
MATERIALS AND METHODS
|
|---|
The study protocol had institutional review board approval. Written informed consent for the mammogram reading study and eye movement recording was obtained from all participants. Patient informed consent for use of the mammograms that were collected from 1998 to 2000 was not required by the institutional review board. Although the mammograms were selected before the Health Insurance Portability and Accountability Act went into effect, the selection and coding of the mammograms to ensure anonymity were in compliance with Health Insurance Portability and Accountability Act standards.
Selection and Display of Mammograms
The test set consisted of craniocaudal (CC) and mediolateral oblique (MLO) views from 20 normal and 20 abnormal screen-film mammograms that were collected as part of a previous study (2). The abnormal mammograms depicted lesions that were not observed during usual clinical screening by Mammography Quality Standards Act–certified mammographers. A year later, the lesions were reported and proved to be malignant at core biopsy or excision. The lesions consisted of 15 masses, three calcifications, one architectural distortion, and one mass with architectural distortion. Because the cancers were not observed although they were visible retrospectively, the mammograms were considered to be "difficult." The cancer-free control mammograms were selected by a mammographer (S.P.W.) from a pool of studies with overall characteristics similar to those of the cancer-containing studies; the patients in whom the control mammograms were obtained had negative mammographic results for a minimum of 2 years of follow-up. The mammograms were digitized to 50-µm pixel resolution with a digitizer (Lumiscan Model 100; Lumysis, Sunnyvale, Calif). The CC and MLO views were displayed side by side at a resolution of 127 µm per pixel on a 21-inch landscape monitor with a spatial resolution of 2560 x 2048 (DS5000L; Clinton Electronics, Rockford, Ill). The CC view was always on the observer's left side, with the MLO view on the right, as shown in Figure 1. The monitor had a usable intensity range of 2–340 candela per square meter and was calibrated by using a perceptually tempered gray scale (9).
View larger version (96K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 1: Scan path of mammographer viewing a mammogram with a mass in CC (left) and MLO (right) views. Upper left: First 5 seconds of scan path. Viewing started in center of display between CC and MLO views. The gaze is immediately directed to the mass in the CC view with a long saccade followed by a circumferential scan of the CC view. Upper right: Scan continues to 13 seconds; there is a long saccade to the mass in the MLO view followed by a circumferential scan of the MLO view. Lower left: From 13 to 21 seconds, a series of comparison saccades were made. Lower right: Finally, the last 10 seconds of viewing repeats the circumferential surveys of the CC and MLO views.
|
|
Observers and Viewing Procedure
Three mammographers (including E.F.C. and S.P.W.), one attending radiologist who regularly interpreted mammograms, two mammography fellows, and three radiology residents on rotation in the mammography section participated. They viewed the mammograms at a distance of 38 cm while the gaze was tracked with a head-mounted eye-head tracker (ASL Model 4000SU; Applied Science Laboratories, Bedford, Mass). Before they viewed the test mammograms, the observers were shown two mammograms to familiarize them with the procedure. The mammograms were shown in a random sequence. The viewing of each mammogram was preceded by a calibration pattern. The observer was instructed to search the mammogram for malignancies, to say "done" when satisfied with the search, and to indicate if the mammogram did or did not contain evidence of malignancy. Eye tracking was discontinued, and the observer pointed with a mouse-driven cursor either to each lesion that was suspicious for malignancy on the CC and MLO views or to the display center—if the observer believed both views were normal with the same level of confidence—and clicked the mouse button.
In the rare situation that the observer wished to indicate a different confidence of normal for each view, the views could be separately selected. At the mouse click, a menu window opened with the terms normal, calcification, mass, and architectural distortion. The observer selected one or more terms, and then a second menu window opened with the terms high, medium, and low to indicate the confidence level for malignancy or for normality. The observer selected one and then went on to either the next lesion or the next study. The cursor location, disease, and confidence were recorded. The observers viewed the entire test set in one session, with the opportunity for rest breaks when they felt fatigued.
Scoring-Decision Outcomes with Mammogram as Unit of Analysis
The numeric values of the confidence levels assigned to the decision "cancer" were high (n = 6), medium (n = 5), and low (n = 4), and those assigned to the decision "normal" were low (n = 3), medium (n = 2), and high (n = 1). A response on one view of a cancer-positive mammogram was scored as true-positive if the cursor location was within 1° (7 mm) of the lesion center. All other indicated locations on cancer images were scored as wrong locations. An overall numeric confidence score based on the responses on the two views was assigned to each study by using the following rules:
1. A study true-positive was assigned if either one or both views of a cancer-positive mammogram were scored true-positive. The highest true-positive confidence score was assigned.
2. A study false-negative was assigned if both views of a cancer-positive mammogram were scored false-negative. The highest numeric false-negative confidence score was assigned to the study.
3. A study false-negative was assigned if both views of a cancer-positive mammogram were scored as wrong locations or if one view was scored as wrong locations and the other was scored as false-negative. A default confidence score of 3 was assigned.
4. A study false-positive was assigned if either one or both views of a control mammogram were scored as false-positive. The highest confidence score was assigned.
5. A study true-negative was assigned if both views of a control mammogram were scored as true-negative. The highest confidence score was assigned.
Definition of Useful Visual Field and Hit Time
The eye-position data consisted of sequential x,y location coordinates digitized at a rate of 60 samples per second. The data samples were reduced to fixations by using methods previously described (10). A gaze scan path consists of a series of dots indicating the center of the fixations connected by straight lines. We assume that inconspicuous objects—those of small size, low contrast, or indistinct boundaries or those embedded in complicated backgrounds—must be viewed by the central vision for recognition to occur. The roughly circular area around the axis of the gaze responsible for recognition is called the useful visual field (UVF) or the conspicuity zone (7,11). The size of the functional UVF depends on both object and background properties (10). The size of the UVF for the cancers in the test set used in this experiment was estimated as a 2.5° radius by using the gaze-tracking data. The fixation data were analyzed by using computer programs developed by one of the authors (H.L.K.). The programs calculated the euclidean distance in degrees of visual angle between the center of each of the lesions that were correctly reported and the x,y coordinates of the fixations that came closest to the lesion center. The elapsed time from the onset of viewing until the lesion center lies within the boundaries of the UVF is called the time to first hit for the lesion, or simply "hit time."
Hit Times for Random Targets in Control Mammograms
As a control for the hit times on cancer-positive mammograms, hit times were calculated for randomly selected locations on the cancer-free mammograms that were scored as true-negative. A random number generator constrained to generate an x,y coordinate within the soft-tissue boundaries of the breast image was used to establish a location in the CC and MLO views. The gaze scan path for each true-negative trial was used to compute the median hit time for each observer as the average of 100 pairs of random locations for each true-negative trial.
Analysis of Initial Saccades
The analysis of the initial saccades was performed by determining if the cancer was hit within the first three saccades and if the first saccade was longer than 15°—equivalent to three UVF diameters. The three-saccade tolerance was chosen because many initial long saccades miss the target but are immediately followed by short "corrective" saccades to the target. The situation is illustrated by A in Figure 2, where an initial long saccade is followed by short saccades to the lesion. B in Figure 2 illustrates an initial long saccade without subsequent fixation of the lesion, and C in Figure 2 illustrates a hit without a long saccade. Neither B nor C would be considered to show initial long saccades to the lesion.
View larger version (9K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 2: Rules used to score initial saccades. X indicates starting position, star indicates cancer, arrows indicate saccades, and circles show a 5°-diameter UVF for reference. A shows a long saccade (greater than three UVFs) followed by two short saccades to the lesion. This is scored as an initial long saccade to the lesion. B shows a long saccade (greater than three UVFs) followed by two short saccades that do not reach the lesion. This is scored as an initial long saccade but not as a long saccade to the lesion. C shows a hit within three saccades, but the first saccade is only two and a fraction of a UVF long. The initial saccade is not a long saccade, and this situation does not qualify as a long saccade to the lesion.
|
|
Statistical Analysis
Receiver operating characteristic analysis was performed with the mammogram as the unit of analysis. The computer program ROCKIT (12) was used to estimate the area under the receiver operating characteristic curve (Az) for each observer. The Az values for each observer were transformed into an index of detectability, da, by using da = z(Az) · 
2, where z is the normal deviate (13). Linear regression, Wilcoxon, and t tests were performed by using software (StatView, version 5.0, 1998; SAS Institute, Cary, NC). The differences between the hit times were tested by using the Wilcoxon test because the distribution was not normal. A P value of .05 was considered to indicate that the observed difference was statistically significant.
|
RESULTS
|
|---|
Expertise can be expressed in terms of the level of training or as the number of studies read in a specified time interval (14). For the purpose of this analysis, the Az value determined from the decision performance on the test set was used because it is a measurable quantity. We found a very wide range in performance on this test set (Table 1).
Hit Time
The average search time on the 20 cancer-positive mammograms for all observers was 26.9 seconds, with a range of 18.2–39.4 seconds. The median time for the entire group to hit a cancer regardless of the decision outcome was 1.13 seconds (Table 2). The hit times for the entire group for cancers on mammograms scored as true-positive were considerably shorter (0.87 second) than those for mammograms scored as false-negative (2.37 seconds). The Wilcoxon test revealed that the difference between true-positive and false-negative was statistically significant (z = 3.83, P < .001).
The time required to first hit the cancers was correlated with performance as expressed by da. The squared correlation coefficient (r2) was 0.81 (Fig 3). The correlation is stronger (r2 = 0.87) if only the true-positive outcomes are considered.
View larger version (9K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 3: Graph shows relationship of median time (in seconds) to first hit a cancer with the UVF and the ability to identify a cancer in the test set as expressed by the receiver operating characteristic index of detectability, da. Individual data points and regression line are shown.
|
|
Regarding the median hit times averaged over 100 pairs of random targets for each mammogram scored as true-negative (Table 3), the Wilcoxon test revealed that the difference between true-positive (0.87 second) and true-negative (3.88 seconds) was statistically significant (z = 13.40, P < .001), as was the difference between false-negative (2.37 seconds) and true-negative (3.88 seconds) (z = 7.73, P < .001). The average search time on the 20 control mammograms for all observers was 29.7 seconds, with a range of 19.2–41.7 seconds. On average, 89% of the random locations placed on the true-negative mammograms were hit in a 5°-diameter UVF (Table 3), indicating that there was relatively complete coverage of the control mammograms by this UVF in approximately 30 seconds of viewing.
View this table:
[in this window]
[in a new window]
|
Table 3. Number of Control Mammograms Included in Random Location Analysis, Percentage of Random Locations Hit by a Fixation, and Median First Hit Time
|
|
Global Analysis
Evidence for the localization of a potential lesion through global analysis of the image was obtained from analysis of the first long saccade (Fig 1) after the initial fixation in the center of the display, a point midway between the CC and the MLO views. Regarding the average percentage of long saccades made initially by each observer and the percentage of initial long saccades aimed at both the cancers and the randomly assigned locations (Table 4), a paired t test indicated no significant difference between the proportion of initial long saccades in the cancer group and that in the control group (P = .7) but a highly significant difference between the proportion of initial saccades aimed at the cancer and those aimed at a randomly selected location on a control mammogram (P < .001).
View this table:
[in this window]
[in a new window]
|
Table 4. Total Number of Initial Long Saccades and Number of Initial Long Saccades Aimed either Directly at a Cancer or at a Random Location in a Control Case
|
|
Review of the scan paths showed that 40% (72 of 180) of the scans of the cancer-positive mammograms were similar to the one shown in Figure 1, with an initial long saccade to either a true lesion or some other perturbation in the image, followed by circumferential scanning. The three observers with the best decision performance accounted for 23% (41 of 180) of the global-type scans. The CC view was scanned first in 83% (299 of 360) and 78% (280 of 360) of the trials on the cancer and control mammograms, respectively. Ninety-one percent of the lesions were first hit on the CC view, even though the lesion in the MLO view was usually closer to the starting point of the scan in display center. The average distance of the true lesions from the center was 23.2° (range, 16.1°–28.6°) for the CC view and 9.9° (range, 3.4°–18.9°) for the MLO view. A total of 62% (112 of 180) of the scans on the control mammograms followed the pattern of a circumferential scan of one view—usually the CC—followed by a similar scan of the other view (Fig 4).
View larger version (38K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 4: Examples of gaze scan paths for first 10 seconds of viewing of a mammographer (left), fellow (middle), and resident (right) looking at the same cancer-free two-view mammogram (left: CC view, right: MLO view). Notice the general similarity of the patterns and the tendency to complete a circumferential scan of one view before jumping to the other one. There are subtle differences in the scan paths in that by 10 seconds the mammographer has covered most of the image while the resident has not scanned the superior aspect of the MLO view.
|
|
|
DISCUSSION
|
|---|
Half of the subtle cancers in the test set of our study were identified and fixated in about 1 second of viewing. This allows insufficient time for a complete search of the mammogram with the central vision. The best observers actually jumped directly to the cancer on first seeing the image. These data support the hypothesis that an initial global image analysis produces a holistic perception that enables the rapid identification of abnormalities and that the ability to utilize the information in the holistic perception improves with diagnostic proficiency.
The difference between the best and the worst decision performance as measured by Az is not a matter of incomplete or premature cessation of search but is a matter of both an inability to holistically identify perturbations that on closer examination will turn out to be cancers and an inability to recognize cancers even when they are fixated. The observer with the highest Az identified the location of 55% (11 of 20) of cancers before first fixating them with a long saccade. The observer with the lowest Az only identified 20% (four of 20) of the cancers but actually fixated 90% (18 of 20) of the cancers during the subsequent search-to-find mode, recognizing only two of them.
The fact that initial recognition of a cancer takes place so rapidly suggests that a perceptual process capable of analyzing the input of the entire retina must be operating. This process is not only capable of characterizing the complete radiologic scene but is also capable of pinpointing the spatial location of the feature causing the perturbation within the scene (15). Past research has shown that experienced radiologists can accurately recognize many abnormalities on chest images in a single glance (4,16,17). The current study is limited by the use of a small, highly selected set of test mammograms and a small group of observers. The results cannot be generalized to observer subgroups without accounting for native ability, acquired knowledge, practical experience, and deliberate practice.
There is a growing body of evidence from research in psychology that indicates that facial recognition occurs holistically and that holistic recognition may extend to other scenes (3,18). Results of recent neurophysiologic studies (19–21) have also shown that the primate brain has neural mechanisms for holistic recognition.
Holistic perception may also play a key role in the development of expertise in radiographic interpretation. Our results indicate that less-expert observers are unable to draw on the initial holistic perception of the mammogram for signals about the location of cancers. Without the help of cues from the global overview, they are left to search the image with central vision to discover image perturbations and they must depend on local feature analysis to decide if a perturbation is a true abnormality. The exclusive use of the search-to-find strategy leads to slower identification and more errors. Experienced mammographers do not abandon the search-to-find strategy but use it after holistic recognition, frequently with magnification. Myles-Worsley et al (4) surmised that radiologic expertise depends on two kinds of knowledge: "knowledge of the characteristic features of clinically normal exemplars of a class of x-ray films (eg, chest films), and knowledge of the particular set of uncharacteristic features that signal pathology." The mammographer's knowledge base, particularly knowledge of the normal, is built up by viewing large numbers of mammograms. As Wood (22) has pointed out, a dedicated mammographer interprets many thousands of mammograms per year and "synthesizes them into a coherent, organized and searchable mental matrix of diagnostic meaning and pathologic features." Deliberate practice enhances perceptual learning and enables observers to migrate from exclusive search-to-find strategies to more efficient holistic strategies in which the perceptual mechanism for the recognition of an abnormal mammogram is similar to one used for the recognition of a familiar face (23,24).
|
ADVANCES IN KNOWLEDGE
|
|---|
- Evidence is presented for the existence in trained observers of a rapid global recognition process that leads to a holistic perception and identification of the location of cancers in mammograms in less than 1 second of viewing.
- Holistic perception and recognition precedes the search-to-find mode of mammogram viewing and is more efficient in proficient observers.
|
FOOTNOTES
|
|---|
Abbreviations: Az = area under receiver operating characteristic curve • CC = craniocaudal • MLO = mediolateral oblique • UVF = useful visual field
Authors stated no financial relationship to disclose.
Author contributions: Guarantor of integrity of entire study, H.L.K.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, H.L.K., C.F.N.; clinical studies, C.F.N.; experimental studies, all authors; statistical analysis, H.L.K., C.F.N.; and manuscript editing, C.F.N., E.F.C., S.P.W.
|
References
|
|---|
- Christensen EE, Murry RC, Holland K, et al. The effect of search time on perception. Radiology 1981;138:361–365.[Abstract/Free Full Text]
- Nodine CF, Mello-Thoms C, Kundel HL, Weinstein SP. The time course of perception and decision making during mammographic interpretation. AJR Am J Roentgenol 2002;179:917–923.[Abstract/Free Full Text]
- Wallis G, Bulthoff H. Learning to recognize objects. Trends Cogn Sci 1999;3:22–31.[CrossRef][Medline]
- Myles-Worsley M, Johnston WA, Simons MA. The influence of expertise on X-ray image processing. J Exp Psychol Learn Mem Cogn 1988;14:553–557.[CrossRef][Medline]
- Nodine CF, Kundel HL, Lauver SC, Toto LC. Nature of expertise in searching mammograms for breast masses. Acad Radiol 1996;3:1000–1006.[CrossRef][Medline]
- Kundel HL, Nodine CF, Thickman DI, Toto LC. Searching for lung nodules: a comparison of human performance with random and systematic scanning models. Invest Radiol 1987;22:417–422.[CrossRef][Medline]
- Findlay JM. Saccade target selection during visual search. Vision Res 1997;37:617–631.[CrossRef][Medline]
- Schall JD. Neural basis of saccade target selection. Rev Neurosci 1995;6:63–85.[Medline]
- Kundel HL, Weinstein SP, Conant EF, Toto LC, Nodine CF. A perceptually tempered display for digital mammograms. RadioGraphics 1999;19:1313–1318.[Abstract/Free Full Text]
- Nodine CF, Kundel HL, Toto LC, Krupinski EA. Recording and analyzing eye-position data using a microcomputer workstation. Beh Res Meth Inst Comp 1992;24:475–485.
- Kundel HL, Nodine CF, Toto L. Searching for lung nodules: the guidance of visual search. Invest Radiol 1991;26:777–781.[Medline]
- ROCKIT. University of Chicago, 2004. http://www-radiology.uchicago.edu/krk/KRL_ROC/software_index.htm. Accessed October 8, 2005.
- Macmillan NA, Creelman CD. Detection theory: a user's guide. Cambridge, UK: Cambridge University Press, 1991.
- Esserman L, Cowley H, Eberle C, et al. Improving the accuracy of mammography: volume and outcome relationships. J Natl Cancer Inst 2002;94:369–375.[Abstract/Free Full Text]
- Swensson RG. A two-stage detection model applied to skilled visual search by radiologists. Percept Psychophys 1980;27:11–16.
- Kundel HL, Nodine CF. Interpreting chest radiographs without visual search. Radiology 1975;116:527–532.[Abstract]
- Oestmann JW, Greene R, Kushner DC, et al. Lung lesions: correlation between viewing time and detection. Radiology 1988;166:451–453.[Abstract/Free Full Text]
- Schyns PD, Gosselin F. Diagnostic use of scale information for componential and holistic recognition. In: Peterson MA, Rhodes G, eds. Perception of faces, objects and scenes. Oxford, England: Oxford University Press, 2003; 120–145.
- Bichot NP, Rossi AF, Desimone R. Parallel and serial neural mechanisms for visual search in macaque area V4. Science 2005;308:529–534.[Abstract/Free Full Text]
- Wolfe JM. Watching single cells pay attention. Science 2005;308:503–504.[Abstract/Free Full Text]
- Wolfe JM. Guided search 2.0: a revised model of visual search. Psychol Bull Rev 1994;000:202–238.
- Wood BP. Visual expertise. Radiology 1999;211:1–3.[Free Full Text]
- Sowden PT, Davies IR, Roling P. Perceptual learning of the detection of features in x-ray image: a functional role for improvements in adults' visual sensitivity? J Exp Psychol Hum Percept Perform 2000;26:379–390.[CrossRef][Medline]
- Charness N, Krampe R, Mayr U. The role of practice and coaching in entrepreneurial skill domains: an international comparison of life-span chess skill acquisition. In: Ericsson KA, ed. The road to excellence. Mahwah, NJ: Lawrence Erlbaum Associates, 1996; 51–80.
TOP
ABSTRACT
INTRODUCTION
