Radiology
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

DOI: 10.1148/radiol.2411050792
This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Woo, J. H.
Right arrow Articles by Melhem, E. R.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Woo, J. H.
Right arrow Articles by Melhem, E. R.
(Radiology 2006;241:206-212.)
© RSNA, 2006

Neuroradiology

Detection of Simulated Multiple Sclerosis Lesions on T2-weighted and FLAIR Images of the Brain: Observer Performance1

John H. Woo, MD, Lana P. Henry, MD, Jaroslaw Krejza, MD, PhD2 and Elias R. Melhem, MD

1 From the Department of Radiology, Division of Neuroradiology, Hospital of the University of Pennsylvania, 3400 Spruce St, Dulles 2, Philadelphia, PA 19104. From the 2004 RSNA Annual Meeting. Received May 9, 2005; revision requested July 7; revision received October 9; accepted November 14; final version accepted December 22. Address correspondence to J.H.W. (e-mail: woojohn{at}uphs.upenn.edu).


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
Purpose: To determine observer performance in the detection of multiple sclerosis (MS) lesions on magnetic resonance (MR) images of the brain and to assess the dependence of observer performance on lesion size, parenchymal location, pulse sequence, and supratentorial versus infratentorial level.

Materials and Methods: This HIPAA-compliant protocol was approved by the institutional review board, and previously acquired MR data from a healthy volunteer and a patient with MS were used to derive parameter maps, with waiver of informed consent. Parameter maps and image simulator software were used to generate 320 phantom brain images with simulated supratentorial and infratentorial MS lesions. Images were displayed with T2-weighting or fluid-attenuated inversion recovery (FLAIR) contrast. Four readers independently evaluated the images, rating lesions on a five-point certainty scale. Observer performance was measured by using the area under the alternative free-response receiver operating characteristic curve (A1), and significance was determined with the z test.

Results: Pooled A1 scores were significantly better for FLAIR imaging (0.96 ± 0.01 [standard error]) than for T2-weighted MR imaging (0.89 ± 0.04) supratentorially (P = .05) but were similar for FLAIR imaging (0.90 ± 0.06) and T2-weighted MR imaging (0.88 ± 0.05) infratentorially. A1 scores for cortical, deep white matter, and periventricular lesions were 0.93 ± 0.05, 0.97 ± 0.02, and 0.89 ± 0.04, respectively, for FLAIR imaging and 0.77 ± 0.06, 0.99 ± 0.01, and 0.89 ± 0.05, respectively, for T2-weighted MR imaging. FLAIR scores were significantly higher than T2-weighted scores for cortical lesions. Linear correlation was found between A1 and lesion size (r = 0.5).

Conclusion: Supratentorially, performance was better with FLAIR imaging than with T2-weighted MR imaging. Infratentorially, performance was moderate with both modalities. Observers did better with FLAIR imaging in the detection of cortical lesions, and performance improved with increasing lesion size.

© RSNA, 2006


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
Magnetic resonance (MR) imaging of the brain plays a crucial role in the imaging of multiple sclerosis (MS). However, it is difficult to obtain objective measures of observer performance in the task of MS lesion detection on MR images. Lesions are (in general) easily seen, particularly on T2-weighted and fluid-attenuated inversion recovery (FLAIR) MR images, but pathologic correlation with tissue sampling is rarely, if ever, performed. As a result, any attempt to design a study by using real MR images to measure observer performance in the detection of MS lesions would be impeded by this lack of correlative truth data.

As a partial solution to this problem, previous investigators used an MR image simulator to create phantom images with simulated lesions (1,2). Herskovits et al (1) tested observers to assess how performance depended on lesion location and MR sequence. Their testing paradigm was limited, however, by the forced binary nature of the observer response—a lesion was either present or absent. As a result, their analysis yielded only sensitivity and specificity values as measures of performance. Moreover, they did not test observers with images of the posterior fossa, where many researchers believe that MS lesions are less detectable on FLAIR images (3,4). Thus, the purpose of our study was to determine observer performance in the detection of MS lesions on MR images of the brain and to assess the dependence of observer performance on lesion size, parenchymal location, pulse sequence, and supratentorial versus infratentorial level.


   MATERIALS AND METHODS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
The details of the MR simulator have been previously published (1,2). A specialized acquisition sequence was used to derive MR parameter maps of the brain in a healthy 40-year-old male volunteer who had no known neurologic disease. These images were judged to be normal by all four authors (0–12 years of experience in diagnostic neuroradiology). Parameter maps were similarly acquired in a 32-year-old woman with MS. Both participants had previously given written informed consent to a prior study that was approved by the institutional review board. For the current study, which was also approved by the institutional review board, informed consent was waived. The study was compliant with the Health Insurance Portability and Accountability Act because only the MR data were used, with all the identifying information deleted.

Template Data
A 1.5-T MR imager (ACS-NT; Philips Medical Systems, Best, the Netherlands) was used to perform mixed multiecho spin-echo and inversion-recovery MR imaging in the healthy volunteer. This sequence yielded eight spin-echo MR images (repetition time, 1500 msec) with different echo times (20, 40, 60, 80, 100, 120, 140, and 160 msec) and eight inversion-recovery MR images (repetition time, 2000 msec; inversion time, 400 msec) with the same eight echo times. Section thickness was 5 mm, in-plane resolution was 0.80 x 0.86 mm (rectangular field of view, 165 x 220 mm; matrix, 205 x 256), and acquisition time was 9 minutes 30 seconds. Pixel maps (256 x 256) of T1 relaxation rate, T2 relaxation rate, and proton density were generated online (software release 6.2; Philips Medical Systems) by using the multiecho data. The multiecho sequence was performed twice, with images obtained in the brain at two transverse levels (supratentorial and infratentorial). The supratentorial level included the lateral ventricles at the septum pellucidum, and the infratentorial level intersected the fourth ventricle and orbital floors.

Lesion Data
By performing the same multiecho MR sequence on the patient with MS, we obtained similar parameter maps for a lesion in the left centrum semiovale. All four authors agreed on the presence of this lesion. The lesion was digitally extracted by including only those pixels that had T1, T2, and proton density values that were more than 2 standard deviations above the mean value in the contralateral normal-appearing white matter, which resulted in a lesion diameter of 7 pixels. This threshold of 2 standard deviations above the mean was previously shown to separate the lesion from the surrounding white matter (2). Resampling with bicubic interpolation generated smaller lesions that were 2, 3, 4, 5, or 6 pixels in diameter.

MR Image (Test Case) Generation
The MR image simulator software used a steady-state solution to the Bloch equation and calculated signal intensity (S) as a function of three tissue parameters (ie, proton density [{rho}], T1, and T2) and three machine parameters (repetition time [TR], echo time [TE], and inversion time [TI]) at the x and y coordinates:

Formula
This is the same formula used in previous implementations of the MR simulator (1,2), with an additional term in the denominator that better approximates steady-state signal intensity (5). By applying this formula on a pixel-by-pixel basis, we generated images with either T2-weighting (4500/100 [repetition time msec/echo time msec]) or FLAIR contrast (11 000/140/2600 [repetition time msec/echo time msec/inversion time msec]). A lesion was embedded in a template by choosing the greater of the signal intensities computed for the lesion and normal brain tissue.

A total of 80 supratentorial images were first generated, with 20 images each containing zero, one, two, or three lesions. Thus, a total of 120 lesions were placed. These lesions were deliberately chosen in order to represent equally each of the five lesion sizes (2, 3, 4, 5, or 6 pixels), with 24 lesions of each size. An approximately equal number of lesions was also distributed among cortical, deep white matter, and periventricular locations, with about 40 lesions in each location. This procedure was repeated at the infratentorial level, with similar attention given to the distribution of sizes and locations, thereby creating 80 more images. Each of the resulting 160 images was used twice (once with T2-weighting and once with FLAIR contrast), yielding 320 total test cases (Fig 1). All 320 images were shown to each reader.


Figure 1
View larger version (93K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1a: Representative simulated transverse FLAIR (11000/140/2600) and T2-weighted (4500/100) MR images show lesions in same location. (a) At the supratentorial level, the callosal lesion (arrow) is seen equally well on T2-weighted (right) and FLAIR (left) images, but the cortical lesion (arrowhead) is much more visible on the FLAIR image. (b) At the infratentorial level, the two lesions are seen equally well on T2-weighted (right) and FLAIR (left) images, with one lesion (arrowhead) in the pontine white matter and the other (arrow) in the periventricular white matter near the fourth ventricle.

 

Figure 1
View larger version (104K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1b: Representative simulated transverse FLAIR (11000/140/2600) and T2-weighted (4500/100) MR images show lesions in same location. (a) At the supratentorial level, the callosal lesion (arrow) is seen equally well on T2-weighted (right) and FLAIR (left) images, but the cortical lesion (arrowhead) is much more visible on the FLAIR image. (b) At the infratentorial level, the two lesions are seen equally well on T2-weighted (right) and FLAIR (left) images, with one lesion (arrowhead) in the pontine white matter and the other (arrow) in the periventricular white matter near the fourth ventricle.

 
Testing
Our experiment followed a multiple-reader multiple-case design. Four board-certified neuroradiologists, each in their 1st year of diagnostic neuroradiology fellowship training, evaluated the images by using testing software written in Interactive Data Language (Research Systems, Boulder, Colo). None of the four readers were authors. Each reader was instructed to find and designate all suspected lesions on the images by using a point-and-click mouse interface (described below). Before actual testing began, the reader was allowed to practice with this interface during a demonstration session. The reader was informed that each image could contain any number of lesions, including zero, with no maximum specified. Each testing session occurred at the same workstation with a 21-inch monitor, and window width and center levels were chosen to match those of standard MR images of the brain.

The 320 images, each linearly interpolated to a 512 x 512 matrix, were displayed one at a time in a random order, alternating between supratentorial and infratentorial images. After detecting a possible lesion, the reader used the mouse to locate the lesion with a button click. A drop-down menu then allowed the reader to assign one of the following certainty ratings: 4, definite lesion; 3, probable lesion; 2, possible lesion; and 1, cannot exclude lesion. These ratings reflected a decreasing scale of reader certainty regarding the presence of a lesion so that, for example, a rating of 1 (cannot exclude lesion) was less certain than a rating of 2 (possible lesion). The location and rating score were recorded for each response. If desired, the reader could delete a response with a different button click. This functionality was allowed, for example, in case the reader mistakenly clicked at an unintended location or if the reader simply reconsidered a prior response.

The reader was asked to locate and score all lesions that were found on the image within a 30-second time limit. A visual signal indicated when 25 seconds had elapsed, and at 30 seconds the image would be erased from the screen. If satisfied that all the lesions were found, the reader could proceed to the next image before the time limit expired. Breaks were allowed between images, and each reader was allowed to take the test in two separate 160-image sessions to reduce fatigue.

Scoring
The tests were scored by using the alternative free-response receiver operating characteristic (AFROC) scoring method (6). A response was deemed true-positive if located within 5 pixels (in both x and y coordinates) of a true lesion. Each true-positive response was assigned a "hit" score of 1–4, depending on the reader's certainty (1 for least certain to 4 for most certain). An undetected lesion (ie, a false-negative finding) was assigned a hit score of 0. In this way, each lesion (detected or not) was assigned a hit score of 0–4.

Any response not within 5 pixels of a true lesion was deemed a false-positive response. For each image, the false-positive image score recorded the maximum rating of all false-positive responses for that image. For example, if two false-positive responses were given, with certainty scores of 1 and 2, then the resulting false-positive image score for that image would be 2. An image with no false-positive responses was assigned a false-positive image score of 0 (true-negative result). In this way, each image was assigned a false-positive image score of 0–4.

For each reader, this scoring method enabled the construction of an AFROC curve (6), which is analogous to the receiver operating characteristic curve that is used in traditional receiver operating characteristic studies. This curve plotted the true-positive fraction of lesions versus the false-positive fraction of images. Observer performance was quantified as the area under the AFROC curve (A1), which was estimated by using trapezoidal integration. The four A1 values were averaged together to form a pooled metric of observer performance.

Statistical Analysis
We applied the z test to assess for statistical significance at a P level of .05 for comparing performance by using a statistical software program (Excel 2000; Microsoft, Redmond, Wash). To apply the z test for comparing pooled metrics of observer performance, we needed to estimate the appropriate means and standard errors. Mean performance was estimated by using the average of the four A1 values of the four individual readers. Then, as outlined by Hanley (7), the overall standard error could be estimated by using a function that effectively sums the error components caused by interreader and intercase variability. Interreader variability was estimated by using the sample variance of the four individual A1 values. Intercase variability was estimated with a jackknife approach as the sample variance of the resulting pseudovalues calculated by the JAFROC software (version 1.00; www.devchakraborty.com) (8).

We tested for significant differences in performance with respect to modality (FLAIR vs T2-weighted MR imaging) at the supratentorial and infratentorial levels by using the z test. We also applied the z test to evaluate differences in performance (pooled across supratentorial and infratentorial levels) in the detection of lesions located at cortical, deep white matter, and periventricular locations. Finally, we evaluated the correlation coefficient in order to see how overall performance (pooled across both modalities and both levels) varied with lesion size.


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
At the supratentorial level, observer performance was excellent on FLAIR images (pooled A1 score, 0.96 ± 0.01 [standard error]) and only moderate on T2-weighted MR images (pooled A1 score, 0.89 ± 0.04). AFROC curves for the four readers and pooled performance values are plotted in Figure 2a (for T2-weighted MR images) and 2b (for FLAIR images). The difference in performance between modalities was statistically significant (z = 1.97, P = .05).


Figure 2
View larger version (15K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2a: AFROC curves for supratentorial (a) T2-weighted and (b) FLAIR images. Individual AFROC curves for the four observers (R1, R2, R3, and R4) are plotted as dashed lines; the pooled AFROC curve is plotted as a solid line. In a, overall performance in lesion detection is good. Compared with performance in a, performance in b is improved. The y-axis shows the true-positive fraction of lesions, and the x-axis shows false-positive fraction of images.

 

Figure 2
View larger version (13K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2b: AFROC curves for supratentorial (a) T2-weighted and (b) FLAIR images. Individual AFROC curves for the four observers (R1, R2, R3, and R4) are plotted as dashed lines; the pooled AFROC curve is plotted as a solid line. In a, overall performance in lesion detection is good. Compared with performance in a, performance in b is improved. The y-axis shows the true-positive fraction of lesions, and the x-axis shows false-positive fraction of images.

 
At the infratentorial level, pooled A1 scores showed moderate performance on FLAIR images (0.90 ± 0.06) and T2-weighted MR images (0.88 ± 0.05). Corresponding AFROC curves are plotted in Figure 3a (for T2-weighted MR images) and 3b (for FLAIR images). This difference was not statistically significant (P = .76).


Figure 3
View larger version (14K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3a: AFROC curves for infratentorial (a) T2-weighted and (b) FLAIR images. Individual AFROC curves for the four observers (R1, R2, R3, and R4) are plotted as dashed lines; the pooled AFROC curve is plotted as a solid line. In a, overall performance in lesion detection is good. Performance is equivalent in a and b. The y-axis shows the true-positive fraction of lesions, and the x-axis shows false-positive fraction of images.

 

Figure 3
View larger version (14K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3b: AFROC curves for infratentorial (a) T2-weighted and (b) FLAIR images. Individual AFROC curves for the four observers (R1, R2, R3, and R4) are plotted as dashed lines; the pooled AFROC curve is plotted as a solid line. In a, overall performance in lesion detection is good. Performance is equivalent in a and b. The y-axis shows the true-positive fraction of lesions, and the x-axis shows false-positive fraction of images.

 
Performance also varied according to lesion location (Fig 4), with scores for the supratentorial and infratentorial levels aggregated together. For cortical lesions, performance was good on FLAIR images (0.93 ± 0.05) but only moderate on T2-weighted MR images (0.77 ± 0.06). For deep white matter lesions, performance was excellent on FLAIR images (0.97 ± 0.02) and T2-weighted MR images (0.99 ± 0.01). For periventricular lesions, performance was moderate on FLAIR images (0.89 ± 0.04) and T2-weighted MR images (0.89 ± 0.05). FLAIR scores were significantly higher than T2-weighted MR imaging scores for cortical lesions (z = 2.06, P = .04) but not for deep white matter (P = .51) or periventricular (P = .93) lesions. We also found a moderate linear correlation (r = 0.5, P < .05) between A1 and lesion size (Fig 5).


Figure 4
View larger version (12K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 4: Observer performance in the detection of MS lesions is significantly better on FLAIR images (white bars) than T2-weighted MR images (black bars) for gray matter (GM) lesions. Performance is similar for deep white matter (DWM) and periventricular (PVWM) lesions. Error bars = standard error.

 

Figure 5
View larger version (12K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 5: Observer performance in the detection of MS lesions on FLAIR and T2-weighted MR images is shown as a function of lesion size. Performance increases with increasing maximum lesion size from 2 to 6 pixels (r = 0.5; P = .04). Error bars = standard error.

 

   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
ADVANCES IN KNOWLEDGE
 References
 
In this study, we applied free-response methods to obtain prospective measures of observer performance in the detection of simulated MS lesions. We found that, supratentorially, performance was significantly better on FLAIR images than on T2-weighted MR images. This finding is in agreement with the findings of Herskovits et al (1) and is also consistent with the results of older studies (3,4,9–13). In many of these older studies, researchers compared modalities by calculating the number or volume of lesions, as determined by the consensus opinion of radiologists (3,4,9–11) or by using semiautomated lesion-detection software (12,13). While such methods can provide some basis for comparison, they do not approximate the usual conditions of image interpretation, and they cannot account for interreader variability. By using a multiple-reader multiple-case design, we provide better objective evidence that FLAIR images allow a meaningful improvement in performance compared with T2-weighted MR images at the supratentorial level.

Performance was equivalent and moderate for the two modalities at the infratentorial level, corroborating the belief that FLAIR imaging performs worse in the posterior fossa (3,4). Various explanations for this phenomenon include the additional artifacts introduced by the FLAIR sequence—for example, cerebrospinal fluid flow artifacts (14) and/or possible differences in the intrinsic MR characteristics of lesions. Indeed, Stevenson et al (15) reported differences in the T1 and T2 parameters of infratentorial MS lesions that may at least partially explain their decreased visibility on FLAIR images. Interestingly, our results demonstrated that FLAIR imaging performed worse infratentorially, even though we did not model these additional artifacts or consider any differences in underlying lesion characteristics. Therefore, an additional factor must be responsible for this performance decrease.

Perhaps the explanation lies in the anatomic differences between the posterior fossa and the cerebrum. Because the main advantage of FLAIR imaging is that it nulls the high signal intensity within the cerebrospinal fluid spaces, one might postulate that the "performance gain" of FLAIR imaging is more accentuated supratentorially, where the ventricles are larger and the sulci are more abundant. This hypothesis, while reasonable, remains unproved.

We found a monotonic decrease in performance with decreasing lesion size, which is an expected and reasonable finding because smaller lesions should be more difficult to detect. We deliberately chose this size range, knowing full well that MS lesions are usually larger, so that our measures likely underestimate the true performance of radiologists in clinical practice. However, by using small sizes, we assessed performance with the most difficult of lesions to detect, thereby heightening any differences between the variables that we tested, such as modality, location, or transverse level. Had we included lesions of larger size, the differences would have been harder to detect because all performance measures would have been high.

We found that observer performance varied with lesion location, with improved performance on FLAIR images in the detection of cortical lesions, which corroborates the results of Herskovits et al (1). Conceptually, this finding makes sense because FLAIR imaging nulls the cerebrospinal fluid signal intensity that might obscure these lesions on T2-weighted MR images. The cortical and subcortical lesion burden has been found to correlate with regional atrophy (10) and with cognitive impairment (9,16), thereby highlighting the importance of lesion detection in this location.

The number of cases used in this study represented a compromise between too many images, which would lengthen the study time and perhaps introduce the confounding element of fatigue, and too few images, which might reduce the power of the study. In a previous study, Obuchowski (17) estimated the number of cases that would be needed for a receiver operating characteristic study to have adequate power in demonstrating a significant difference between diagnostic modalities. For studies with four observers, a high (approximately 0.90) accuracy of techniques, a small (approximately 0.05) difference between modalities, a 1:1 ratio of normal to abnormal cases, and a small interreader variability, Obuchowski estimated that a total of 287 cases would be needed to detect a difference with 80% power. Our study had a smaller ratio (1:3) of normal to abnormal cases and was a free-response design rather than a receiver operating characteristic design, but we believe that 320 cases should supply adequate power to detect any relevant differences.

Our study has several limitations. We used only a single template at either transverse level to generate the test cases. Therefore, we did not account for interindividual variability in the brain or for section to section variability. Moreover, having only a single template may have falsely elevated our performance measures because of recall effect. Although the images were shown in an alternating sequence to reduce this effect, almost certainly the readers, as trained neuroradiologists, would be able to remember previous images as they interpreted subsequent images. False-positive responses would be reduced, because readers may not score a lesion that they might otherwise have questioned simply because they remember it as an artifact from a previous image. This bias, however, would have affected all measurements equally so that comparisons of performance may still be valid.

Our study is also limited by our use of a single MS lesion. Similar to Melhem et al (2), we used the MR characteristics of an actual MS lesion as a model for the simulated lesions. This method contrasts with the work of Herskovits et al (1), wherein the model lesion was artificially constructed by using an octagonal shape and MR parameters that are typically reported in MS plaques. However, MS lesions in general have a widely variable appearance on MR images, which likely reflects the heterogeneous processes, such as demyelination, gliosis, axonal loss, remyelination, inflammation, edema, or some combination thereof, that are involved at the histopathologic level. Because we did not account for interlesion variability, it may be difficult to extrapolate our results to estimate the true observer performance in the detection of all MS lesions. Moreover, the validity of our method to generate gray matter lesions may be questioned because our model lesion was extracted from white matter. However, removing interlesion variability should reduce our resulting variances, perhaps increasing the power of our results in demonstrating significant differences in performance.

Besides the limitations related to image generation, others arise from our observer testing methods. We believe that 30 seconds allows readers adequate time to evaluate each image, but this allotment may not accurately reflect actual reading conditions in which single images are usually evaluated only for a few seconds, perhaps longer if abnormalities are detected. We did allow readers to spend less time if they were satisfied that all lesions were found. Still, it is unclear what effect this would have had on observer performance. On the one hand, by looking at an image longer, readers might find more true lesions than they might have otherwise found, thereby improving performance. On the other hand, they might also find more false-positive lesions, thereby reducing performance. Again, we expect that such biases would affect performance measures equally so that differences in performance may still be applicable.

Finally, the accuracy of our testing methods is limited because of other fundamental differences with standard image interpretation. Our tests used single images, whereas normally readers can assess contiguous images to help decide whether the findings represent actual lesions. Moreover, modern picture archiving and communication systems allow readers to adjust the window settings to improve lesion visibility; our software did not have this feature. Again, because these limitations can be expected to affect performance measures equally, differences may still be detected and valid.

In summary, performance on FLAIR images is excellent and is better than that on T2-weighted MR images supratentorially. Infratentorially, performance is moderate and is similar between the two modalities. Observers performed better on FLAIR images in the detection of cortical lesions, and performance improved with increasing lesion size. We believe that, as a first approximation, our methods provide a useful approach to measure observer performance in the task of MS lesion detection on MR images of the brain. In the future, one might imagine improving the method, perhaps with multiple brain templates, multiple digitized lesions, and testing software that more closely approximated actual viewing conditions.


   ADVANCES IN KNOWLEDGE
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
References
 


   ACKNOWLEDGMENTS
 
We thank Edward H. Herskovits, MD, PhD, for his original work in performance measurements by using an MR simulator and Abbas Jawad, PhD, for his help in statistical analysis.


   FOOTNOTES
 

Abbreviations: A1 = area under the AFROC curve • AFROC = alternative free-response receiver operating characteristic • FLAIR = fluid-attenuated inversion recovery • MS = multiple sclerosis

2 Current address: Bialystok Medical Academy, Bialystok, Poland Back

Authors stated no financial relationship to disclose.

Author contributions: Guarantors of integrity of entire study, J.H.W., L.P.H.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, J.H.W., L.P.H.; experimental studies, J.H.W., L.P.H.; statistical analysis, J.H.W., J.K.; and manuscript editing, all authors


   References
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 

  1. Herskovits EH, Itoh R, Melhem ER. Accuracy for detection of simulated lesions: comparison of fluid-attenuated inversion-recovery, proton density–weighted, and T2-weighted synthetic brain MR imaging. AJR Am J Roentgenol 2001;176:1313–1318.[Abstract/Free Full Text]
  2. Melhem ER, Herskovits EH, Karli-Oguz K, et al. Defining thresholds for changes in size of simulated T2-hyperintense brain lesions on the basis of qualitative comparisons. AJR Am J Roentgenol 2003;180:65–69.[Abstract/Free Full Text]
  3. Gawne-Cain ML, O'Riordan JI, Thompson AJ, Moseley IF, Miller DH. Multiple sclerosis lesion detection in the brain: a comparison of fast fluid-attenuated inversion recovery and conventional T2-weighted dual spin echo. Neurology 1997;49:364–370.[Abstract/Free Full Text]
  4. Filippi M, Yousry T, Baratti C, et al. Quantitative assessment of MRI lesion load in multiple sclerosis. IV. A comparison of conventional spin-echo with fast fluid-attenuated inversion recovery. Brain 1996;119:1349–1355.
  5. Droege RT, Wiener SN, Rzeszotarski MS, Holland GN, Young IR. Nuclear magnetic resonance: a gray scale model for head images. Radiology 1983;148:763–771.[Abstract/Free Full Text]
  6. Chakraborty DP, Winter LH. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology 1990;174:873–881.[Abstract/Free Full Text]
  7. Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging 1989;29:307–335.[Medline]
  8. Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys 2004;31:2313–2330.[CrossRef][Medline]
  9. Moriarty DM, Blackshaw AJ, Talbot PR, et al. Memory dysfunction in multiple sclerosis corresponds to juxtacortical lesion load on fast fluid-attenuated inversion-recovery MR images. AJNR Am J Neuroradiol 1999;20:1956–1962.[Abstract/Free Full Text]
  10. Bakshi R, Ariyaratana S, Benedict RH, Jacobs L. Fluid-attenuated inversion recovery magnetic resonance imaging detects cortical and juxtacortical multiple sclerosis lesions. Arch Neurol 2001;58:742–748.[Abstract/Free Full Text]
  11. Yousry TA, Filippi M, Becker C, Horsfield MA, Voltz R. Comparison of MR pulse sequences in the detection of multiple sclerosis lesions. AJNR Am J Neuroradiol 1997;18:959–963.[Abstract]
  12. Tubridy N, Barker GJ, MacManus DG, Moseley IF, Miller DH. Three-dimensional fast fluid attenuated inversion recovery (3D fast FLAIR): a new MRI sequence which increases the detectable cerebral lesion load in multiple sclerosis. Br J Radiol 1998;71:840–845.[Abstract]
  13. Bastianello S, Bozzao A, Paolillo A, et al. Fast spin-echo and fast fluid-attenuated inversion-recovery versus conventional spin-echo sequences for MR quantification of multiple sclerosis lesions. AJNR Am J Neuroradiol 1997;18:699–704.[Abstract]
  14. Arakia Y, Ashikaga R, Fujii K, Nishimura Y, Ueda J, Fujita N. MR fluid-attenuated inversion recovery imaging as routine brain T2-weighted imaging. Eur J Radiol 1999;32:136–143.[CrossRef][Medline]
  15. Stevenson VL, Parker GJ, Barker GJ, et al. Variations in T1 and T2 relaxation times of normal appearing white matter and lesions in multiple sclerosis. J Neurol Sci 2000;178:81–87.[CrossRef][Medline]
  16. Rovaris M, Filippi M, Minicucci L, et al. Cortical/subcortical disease burden and cognitive impairment in patients with multiple sclerosis. AJNR Am J Neuroradiol 2000;21:402–408.[Abstract/Free Full Text]
  17. Obuchowski NA. Sample size tables for receiver operating characteristic studies. AJR Am J Roentgenol 2000;175:603–608.[Abstract/Free Full Text]




This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Woo, J. H.
Right arrow Articles by Melhem, E. R.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Woo, J. H.
Right arrow Articles by Melhem, E. R.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
RADIOLOGY RADIOGRAPHICS RSNA JOURNALS ONLINE