| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Computer Applications |
1 From the Departments of Radiology (J.M.H., T.M.B., W.L.C., J.H.S., D.G.) and Biostatistics (H.E.R.), University of Pittsburgh, A443 Scaife Hall, Pittsburgh, PA 15261-0001. Received March 9, 1999; revision requested May 12; revision received June 22; accepted August 25. Supported in part by grant no. CA58283 from the National Cancer Institute, National Institutes of Health. Address reprint requests to J.M.H. (e-mail: jherron@neuronet.pitt.edu).
 |
Abstract |
|---|
 TOP Abstract IntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
MATERIALS AND METHODS: A total of 529 radiographs were displayed on a specially constructed view box at three luminance levels (770, 260, and 85 cd/m2) and three resolutions (100-µm, 200-µm, and 400-µm pixels). Each image was reviewed nine times by six radiologists who participated in this study. The abnormalities included nodule, pneumothorax, interstitial disease, alveolar infiltrates, and rib fracture. Negative (normal) radiographs were also included.
REULTS: Receiver operating characteristic curves indicated that the effect of image luminance was greater than that of resolution. The detection of pneumothorax, interstitial disease, and rib fracture showed statistically significant differences (P < .05) due to luminance. The detection of pneumothorax was the only abnormality with a statistically significant difference due to resolution. There was no evidence that luminance was related to image resolution for any of the abnormalities.
CONCLUSION: At a resolution of 400-µm pixels or higher across the field of view and a luminance of 260 cd/m2 or more, primary diagnosis with posteroanterior chest radiographs is not likely to be affected by the quality of display.
Index terms: Diagnostic radiology, observer performance • Images, display • Lung, diseases, 60.916, 60.917 • Lung, neoplasms, 60.31 • Pneumothorax, 60.73 • Ribs, fracture, 471.41
|
Introduction |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
The effects of image resolution on observer performance have been investigated by using both direct and indirect assessments, but the results to date are inconclusive (2,3). These experiments led to a proposal that standards be established for cathode-ray-tube displays that are used for primary diagnosis of radiologic images. Relative nonuniformities, which clearly exist in traditional but brighter displays (eg, a view box), have always been considered to be of secondary importance. In addition to observer performance studies, the results of subjective assessments indicate that luminance conditions, which impose stricter lighting requirements in the reading room, may result in readers' inability to engage in longer reading sessions due to fatigue and an inability to concentrate (4). If picture archiving and communications systems, or PACS, are to succeed in the clinical environment, factors such as resolution and luminance must be taken into account, and possible performance trade-offs between resolution, contrast sensitivity, and luminance must be well understood. The purpose of this study was to examine the combined effects of image resolution and display luminance on observer performance for detection of abnormalities depicted on posteroanterior chest radiographs.
|
MATERIALS AND METHODS |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Readers rated each image with regard to the likelihood that the specified abnormality was present. Management of the reading sessions was completely computerized. Software was used to automatically determine both the order of presentation of study modes and the order of images within each mode. Software was also used to randomize and counterbalance cases for each reader. The computer presented a computerized scoring form for each image being viewed and kept track of which images had been read, accumulating all responses from readers in a database.
Selection of Readers and Prestudy Training
All six readers who participated in the project were certified by the American Board of Radiology and had a minimum of 3 years experience in the interpretation of chest images. They were not aware of the aims of the study. Radiologists who participated in the preparation of the study and case selection did not participate as readers. All participating radiologists received an "instructions for observers" form for review, and the definition of abnormalities was discussed with each reader prior to the study, at which time a variety of "typical" and "subtle" positive and negative cases were demonstrated. Radiologists were also trained in the use of our computerized scoring system during individualized reading sessions. At these sessions, selected cases were assembled, reviewed, and read in a manner similar to that which occurred during the actual study.
Selection of Cases and Controls
In selecting actually positive and actually negative images for this study, we followed procedures that have been used successfully and that we have previously described (5,6). In the present study, 529 high-quality posteroanterior chest radiographs were selected, and the presence of each of the abnormalities of interest was verified.
Our choice of abnormalities was based largely on the imaging characteristics of those abnormalities. We did not attempt to examine all potential abnormalities that may be diagnosed on the basis of chest radiographs. The numbers of occurrences of nodules, pneumothorax, interstitial disease, alveolar infiltrates, and rib fracture were 181, 119, 149, 76, and 77, respectively (some images showed more than one abnormality). One hundred twenty-four images were negative for all five abnormalities. The inclusion of rib fracture and alveolar infiltrates on our scoring form was designed to increase the type of possible reported abnormalities, thereby making the study more compatible with the clinical environment. We believe that the inclusion of such abnormalities made it more difficult for observers to estimate the frequency distribution of other abnormalities in the study (ie, nodules, interstitial disease, and pneumothorax), thereby reducing potential biases that may result from observer expectations.
Abnormalities were confirmed by means of surgical reports (eg, biopsy results) and other source documents (eg, results from previous or follow-up procedures) when available or, in a limited number of cases, results from other imaging modalities and clinical status. Subtle pneumothoraces were confirmed by observing the line shadow disappearing or the pneumothorax increasing in size on a follow-up image. Negative images were selected on the basis of the existence of a minimum disease-free interval following either a negative biopsy result or a negative radiologic finding.
Each image was viewed by two radiologists who assigned it a quality score of acceptable or unacceptable. Only those images that were scored acceptable by both radiologists were used. In addition, with a priori knowledge of the abnormalities on the original conventional image, all printed images were reviewed for accurate reproduction.
Digitization of Images
The conventional chest radiographs used for this study were recorded on 14 x 17-inch (35.6 x 43.2-cm) Kodak G films (Eastman Kodak, Rochester, NY) exposed with Lanex screens (Eastman Kodak). Each of the selected radiographs was digitized (film digitizer model 100HR; Lumisys, Sunnyvale, Calif) at a spatial resolution of 100 µm and a contrast sensitivity of 12 bits, to produce a 3,504 x 4,205 x 12-bit matrix. This digitizer had a measured modulation transfer function at the Nyquist frequency (5 line pairs per millimeter) of 0.41 and 0.39 in the horizontal and vertical directions, respectively. The useful densities spanned a range from about 0.3 to 3.0 in optical density (OD). From the 100-µm-resolution digitized films, images were created that had resolution characteristics corresponding to digitization at 200 and 400 µm. These images were produced by means of a "filtering pixel averaging" technique described elsewhere (7), which is capable of effective reproduction of the modulation transfer function of digitization with different spot sizes. Note that this data set was also used in a different study (8) to assess the effect of data compression on observer performance. This latter study included a different group of observers.
Printing of Cases
All the images were printed at full size with a laser printer (Ektascan; Eastman Kodak) on Ektascan EHN film (Eastman Kodak) by using the digitized source data or the reconstructed versions of these data to reduce resolution to predetermined levels. Noise contributed by the laser printer, due to quantum mottle and granularity in the printed film, was substantially less than image noise. The density response of the printed image was corrected by computing and downloading a calibration table to the laser printer. The minimum OD of the film was approximately 0.2 for the present photographic base and emulsion. The maximum density of a printed film was approximately 3.2 OD. The contrast response of the original film was maintained, with the consequence that any density value above the maximum density was compressed to the value of 3.2 OD. The overall digitizer-printer-processor response was determined by comparing the density of the original film with that of the laser-printed film. The variations produced by the film processor were within acceptable limits. The processor used in the study was also used in the clinical department and was subjected to daily quality assurance procedures.
Because the original digitized image had pixel spacing of 100 µm and the printer prints at a pixel pitch of 80 µm, which would have resulted in a 20% minification of these images, the internal interpolation program of the printer was used to scale the printed images to the size of the original images. For each case, the look-up table in the printer was adjusted so that the printed images would match the original conventional radiographs to within 0.08 OD over the 0.25–3.00 OD range.
Display of Images
A 14 x 17-inch (35.6 x 43.2-cm) computer-controlled light box with uniform luminance across the field at selectable levels was assembled by using six horizontal and two vertical hot-cathode type (T8) florescent lamps; direct current was used to eliminate flicker. The polarity of the voltage across each of the lamps was controlled by the computer to prevent migration of mercury to one end of the tube. Multiple diffusing screens and optical feedback made it possible to limit the spatial variation of the luminance to within 5% over 80% of the display surface.
The salient feature of the view box was the ability to vary and maintain the illumination at three distinct luminance levels. This was accomplished by reducing the current to each lamp and maintaining the selected luminance with optical feedback. The system has been described in complete detail elsewhere (9). For quality assurance purposes, the luminance level was measured and recorded for each reading session.
We selected the lowest illumination level in the study to simulate the approximate luminance levels of 1,536 x 2,048-pixel monitors (model GMA 202, Tektronix, Beaverton, Ore; model UHR-4820, Aydin Displays, Horsham, Pa) manufactured at the time. Luminances for these monitors were approximately 25 foot-lamberts (85 candelas per square meter [cd/m2]) with a spot size of 190 µm or smaller. At the time the study was planned, this luminance represented the average brightness of most advanced high-resolution workstations. We chose the highest luminance level to simulate the lower level of the conventional viewing stations found in general radiology reading rooms. The level for an unloaded view box varies from 250 to 400 foot-lamberts (850–1,400 cd/m2) for various manufacturers, and it varies within a model due to the age of the lamp. This corresponds to a maximum luminance range of 170–270 foot-lamberts (580–920 cd/m2) when loaded with film. This also represents the expected brightness level of the next generation of advanced workstations. A 225–foot-lambert (770 cd/m2) level was chosen to be representative of this environment. The choice of the middle level was rather arbitrary and was in the range of the capabilities of the current generation of high-resolution workstations. A loaded level of 75 foot-lamberts (260 cd/m2) seemed to be a reasonable estimate. The American College of Radiology recommends a minimum luminance of 50 foot-lamberts (170 cd/m2) for display workstations used in teleradiology.
Performance of Study
The study consisted of a nine-mode comparison with each of the three luminance levels paired with each of the three resolution levels. Six readers read and rated each of the 529 cases in each of nine modes. (A seventh radiologist who started the readings did not complete the task due to leaving the institution, and the interpretation data acquired by this reader were excluded from the analyses.)
Each reading session included 40 randomized cases displayed in one of the reading modes. During a reading session, observers were presented with a stack of envelopes, each containing one laser-printed film image. The images were arranged in the order of interpretations for that session by means of computer randomization. It should be noted that in this study, we did not compare conventional radiographs with the laser-printed images. Observers reported the results for each case directly on our computerized scoring form by using a computer mouse device.
Five sliding scales were presented, one for each abnormality. The radiologists used the mouse to slide an indicator along a scale from 0% to 100% to indicate the likelihood of the presence of the abnormality, with 0% indicating absolute certainty the abnormality in question was not present on the image, and 100% indicating absolute certainty that it was present. For percentages between 0% and 100%, the radiologist selected a percentage that corresponded to the probability that the abnormality was present. The form additionally included multiple choice ratings about image quality and comfort level, as well as subordinate questions that were presented on the basis of answers to the primary detection questions.
All items on the scoring form required a response before the reader could proceed to the next case. The software automatically prevented any case from being read by the same reader more than once in 4 weeks. Given the large number of cases and the complexity of the reading tasks, our experience indicated that this was sufficient time to ensure that cases were not remembered from session to session. Readers were allowed to spend as much time as desired on each image. The amount of time spent on each case (including display, review, and scoring) was monitored and recorded along with the reader's responses for the image.
Data Analysis
Estimates of the areas under ROC curves were calculated for each abnormality (n = 5), mode (n = 9), and reader (n = 6) (5 x 9 x 6 = 270 values) by using the Wilcoxon method (9,10). With this method, the area is computed by using the trapezoidal method and has been shown to be quite powerful in similar situations (10–12).
It should be noted that the set of normal images was complementary to each abnormal finding, including all images negative for that particular abnormality. For example, an image that showed a nodule but no interstitial disease was considered to be negative for the analysis of observer performance for the detection of interstitial disease. A statistical test of the possible effects of luminance and resolution on observer performance was conducted for each of the abnormalities by using a three-by-three two-way analysis of variance with interaction, where the areas under the ROC curves for different readers were considered as replicates. This procedure is an extension of the method proposed by Thompson and Zucchini (13). The test for interaction enabled us to assess whether the effect of resolution on the area under the ROC curve varied for the different levels of luminance.
We also performed the analysis by using resolution, luminance, and reader in a three-way analysis of variance. Although one cannot test for interaction in this design (ie, whether or not the effect of luminance on the areas of the ROC curves was dependent on the resolution level), the three-way analysis of variance provided better control for the high reader variability. Therefore, this method would tend to indicate that smaller effects of resolution and luminance were statistically significant, as long as their effect was relatively consistent within individual readers.
For any instance where there was a statistically significant difference in the area under the ROC curves for either resolution or luminance levels, we performed a two-way analysis to assess the consistency of the findings for the six readers across the three luminance or resolution levels. The Page test for trend (14) was used to determine whether the area under the ROC curve increased with increasing luminance or resolution levels (a one-sided test). The Page test is a modification of the Wilcoxon test, which weights observations so that they are more sensitive in detecting monotonic trends.
|
RESULTS |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
years to complete. Table 1 summarizes the mean areas under the ROC curves for all six readers for the low (400-µm), medium (200-µm), and high (100-µm) resolutions for each of the five abnormalities. Values in Table 1 are means over all six readers and luminance levels. The P values from the two-way analysis of variance all were greater than .5; thus, there was no statistically significant effect of resolution on diagnostic accuracy for any of the five abnormalities, according to the two-way analysis of variance results.
|
|
|
When the three-way analysis of variance was conducted to determine the effect of resolution on the area under the ROC curve, the differences were statistically significant only for the detection of pneumothorax (P < .01). When pairwise comparisons were made, the 400-µm resolution was significantly different from the 200- and 100-µm levels.
The Page test for trend was conducted for the three luminance levels. Table 3 shows the areas under the ROC curves for the detection of pneumothorax by each reader and for each luminance level averaged over all resolution levels. An increase in the areas under the ROC curves for increasing luminance was apparent for all six readers (P < .001). Although the absolute magnitude of the effect was not as large for some of the readers, a similar trend of increasing area under the ROC curve with increasing luminance was observed for the detection of rib fracture (Table 4, P = .01) and interstitial disease (Table 5, P = .05).
|
|
|
The mean time (± SD) needed by readers to view and score each case ranged from 41 seconds ± 22 to 79 seconds ± 31. When averaged over all readers for a particular mode, these times ranged from 64 seconds ± 13 to 69 seconds ± 14. There were no significant differences (P > .4) in duration for the individual modes.
|
DISCUSSION |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
If our results had been significantly different for a larger number of readers and modes, it would have been possible to provide more conclusive statements about the trade-offs between object contrast, resolution, and luminance. At the same time, the fact that resolution had less of an effect than luminance is an important observation, because, to our knowledge, this is the only large multiobserver, multitask, multipoint diagnostic study performed to date. We believe that there are sufficient experimental data published to indicate that for most primary diagnostic tasks, a background luminance of 75 foot-lamberts (260 cd/m2) or greater is sufficient although not necessarily optimal.
Perhaps as important, the interaction between resolution and luminance found in our study was weak at all levels. With the exception of the low-resolution low-luminance mode, observer performance was not significantly affected for a large number of cases, readers, and diagnostic tasks. We recognize the fact that observers' comfort level will not likely determine the use of alternative display modes for primary diagnosis in the clinical environment. It is important, however, that objective data be available in support of such decisions.
In conclusion, the results of our study indicate that in most instances, a true resolution of 1,024 pixels across the field of view and a monitor brightness of 75 foot-lamberts (260 cd/m2) or greater should be sufficient for primary diagnosis with posteroanterior chest images.
|
Footnotes |
|---|
Author contributions: Guarantor of integrity of entire study, D.G.; study concepts, J.M.H.; study design, D.G., J.M.H., H.E.R.; data acquisition, T.M.B., W.L.C., J.H.S.; data analysis, H.E.R.; statistical analysis, H.E.R.; manuscript preparation, D.G., J.M.H., H.E.R.; manuscript editing and review, D.G., J.M.H.
|
References |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  N Buls, W Shabana, P Verbeek, P Pevenage, and J De Mey Influence of display quality on radiologists' performance in the detection of lung nodules on radiographs Br. J. Radiol., September 1, 2007; 80(957): 738 - 743. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. F. Chiang, L. Jiang, R. Gelman, Y. E. Du, and J. T. Flynn Interexpert Agreement of Plus Disease Diagnosis in Retinopathy of Prematurity Arch Ophthalmol, July 1, 2007; 125(7): 875 - 880. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. F. Branstetter IV Basics of Imaging Informatics: Part 1 Radiology, June 1, 2007; 243(3): 656 - 667. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. L. Zuley, K. M. Willison, E. Bonaccio, D. P. Miller, D. L. Leong, P. J. Seifert, P. Somerville, and S. Destounis Full-Field Digital Mammography on LCD Versus CRT Monitors. Am. J. Roentgenol., December 1, 2006; 187(6): 1492 - 1498. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. J. Doyle, J. Le Fevre, and G. D. Anderson Personal Computer versus Workstation Display: Observer Performance in Detection of Wrist Fractures on Digital Radiographs Radiology, December 1, 2005; 237(3): 872 - 877. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. M. Goo, J.-Y. Choi, J.-G. Im, H. J. Lee, M. J. Chung, D. Han, S. H. Park, J. H. Kim, and S.-H. Nam Effect of Monitor Luminance and Ambient Light on Observer Performance in Soft-Copy Reading of Digital Chest Radiographs Radiology, September 1, 2004; 232(3): 762 - 766. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. R. Fuhrman, C. A. Britton, T. Bender, J. H. Sumkin, M. L. Brown, J. M. Holbert, T. S. Chang, H. E. Rockette, and D. Gur Observer Performance Studies: Detection of Single Versus Multiple Abnormalities of the Chest Am. J. Roentgenol., December 1, 2002; 179(6): 1551 - 1553. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
