Published online before print February 28, 2006, 10.1148/radiol.2383050211
(Radiology 2006;239:238-245.)
© RSNA, 2006
Artificial Multiple Sclerosis Lesions on Simulated FLAIR Brain MR Images: Echo Time and Observer Performance in Detection1
Lana Pikus, MS,
John H. Woo, MD,
Ronald L. Wolf, MD, PhD,
Edward H. Herskovits, MD, PhD,
Gul Moonis, MD,
Abbas F. Jawad, PhD,
Jaroslaw Krejza, MD, PhD and
Elias R. Melhem, MD
1 From the Department of Radiology, Hospital of the University of Pennsylvania, 3400 Spruce St, Dulles 2, Philadelphia, PA 19104 (L.P., J.H.W., R.L.W., E.H.H., G.M., A.F.J., J.K., E.R.M.); Division of Neuroradiology, the Johns Hopkins Medical Institutions, Baltimore, Md (E.H.H., E.R.M.); Division of Biostatistics and Epidemiology, the Children's Hospital of Philadelphia, Philadelphia, Pa (A.F.J.); and Department of Radiology, Medical University of Gdansk, Gdansk, Poland (J.K.). Received February 10, 2005; revision requested April 20; revision received May 3; final version accepted June 3.
Address correspondence to: E.R.M. (e-mail: emelhem{at}rad.upenn.edu).
 |
ABSTRACT
|
|---|
The institutional review board approved the described HIPAA-compliant study, which was performed to prospectively evaluate observer performance in the detection of artificial multiple sclerosis (MS) lesions that were randomly distributed supra- and infratentorially on simulated fluid-attenuated inversion-recovery (FLAIR) magnetic resonance (MR) images obtained at different echo times (TEs). MR parametric maps were derived from mixed multi-echo inversion-recovery images obtained in a 40-year-old healthy male volunteer and in a patient with MS, both of whom gave informed consent. Pseudorandomly distributed artificial MS lesions of varying size, number, and location were equally represented on the FLAIR images (11 000/100–200/2600 [repetition time msec/TE msec/inversion time msec]). Twelve images obtained in both regions at each of 11 TEs spaced 10 msec apart were rated by seven neuroradiologists by using a four-point scale. Observer performance in the detection of MS lesions on the FLAIR images, as estimated by using areas under the alternative free-response receiver operating characteristic curve, was highest and most consistent at the 100-msec TE, both supratentorially (93.0% ± 8.6 [standard error of the mean]) and infratentorially (87.4% ± 10.0).
© RSNA, 2006
|
INTRODUCTION
|
|---|
Magnetic resonance (MR) imaging is having an increasingly important role in the diagnosis, management, and understanding of the pathogenesis of multiple sclerosis (MS). MS lesions often can be detected before clinical symptoms develop (1). Serial MR imaging examinations are now used to determine the total lesion load, monitor disease activity, and assess the effectiveness of drug treatments in clinical trials.
Despite the importance of MR imaging with regard to MS, assessment of the accuracy of MR imaging in the detection of MS plaques in the brain is impeded by the lack of a reference standard to verify the true status of the disease in vivo. Tissue sampling usually is not performed; thus, comparisons of imaging findings with pathologic evaluation results are rarely performed, if they are possible at all. Furthermore, a substantial lag time between imaging assessment and pathologic verification can lead to a paradoxical discrepancy between the imaging and pathologic analysis results. These difficulties can be overcome partially by using simulated MR images on which artificial MS lesions are placed in user-specified locations.
In a previous study (2), a simulator was successfully used to assess observer performance in the detection of artificial MS lesions that were randomly distributed in the brain parenchyma. The results showed that fluid-attenuated inversion-recovery (FLAIR) MR sequences had higher accuracy than intermediate- and T2-weighted sequences in the detection of supratentorially located cortical-subcortical and periventricular lesions. In this and other studies (3–6) in which the role of FLAIR MR imaging in the detection of MS lesions was investigated, different echo times (TEs), including 119, 140, and 150 msec, were used. In addition, one previous study (7) revealed that optimal contrast between the MS lesion and the white matter was achieved at TEs of between 130 and 160 msec. To our knowledge, however, no previous study has addressed the issue of the selection of a TE that would optimize observer performance in the detection of MS lesions on FLAIR images. Thus, the purpose of our study was to prospectively evaluate observer performance in the detection of artificial MS lesions that were randomly distributed supra- and infratentorially on simulated FLAIR images obtained at different TEs.
|
MATERIALS AND METHODS
|
|---|
The details of the MR simulator software have been previously described (2,8). We estimated T1 relaxation, T2 relaxation, and proton-density (
) parameter map values to create template supratentorial and infratentorial images of the brain in a 40-year-old healthy male volunteer who had no clinical signs or symptoms of MS. Similar parametric maps of the MS lesion in a symptomatic 32-year-old female patient were acquired. Both subjects gave informed consent. The study protocol was approved by the institutional review board of the Johns Hopkins Medical Institutions and compliant with the Health Insurance Portability and Accountability Act.
The MS lesion in the symptomatic patient was digitally copied, re-sized, and then embedded into the brain templates, as needed, to create multiple maps with simulated lesions of varying size, number, and location. We were able to produce images with FLAIR contrast from these maps by applying an approximate steady-state solution to the Bloch equation on a pixel-by-pixel basis, depending on the chosen repetition times, TEs, and inversion times. Specialized software was then used to display these images to a reader and automatically record her or his responses.
MR Data Acquisition
MR imaging examinations were performed by using a 1.5-T system (ACS-NT; Philips Medical Systems, Best, the Netherlands) with a maximum gradient capability of 23 mT/m and a slew rate of 103 mT/m/msec and by using a quadrature head coil operating in the receive mode. A mixed multi-echo spin-echo inversion-recovery MR sequence was used to obtain transverse images of the brain at the supratentorial and infratentorial levels in the 40-year-old male volunteer. These images were used to generate template images (Fig 1).
View larger version (146K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 1: Computer-generated FLAIR MR images of the supratentorial region of the brain obtained at various TEs. A, Normal image without MS lesions. Images obtained at TEs of, B, 100 msec, and, C, 120 msec show the same three MS lesions (arrows) in cortical gray matter, in deep white matter, and periventricularly, distributed pseudorandomly. Images obtained at TEs of, D, 150 msec, E, 170 msec, and, F, 200 msec show the same three lesions. Lesion visibility decreases with increasing TE.
|
|
The supratentorial level in the brain included the lateral ventricles and the septum pellucidum. The infratentorial level intersected the mid-fourth ventricle and the orbital floors. The mixed multi-echo spin-echo inversion-recovery sequence simultaneously yielded two image data sets: eight spin-echo images (1500-msec repetition time, one signal acquired) and eight inversion-recovery images (2000-msec repetition time, 400-msec inversion time, one signal acquired). Both image sets were obtained at TEs of 20, 40, 60, 80, 100, 120, 140, and 160 msec. For these acquisitions, a section thickness of 5 mm, an in-plane resolution of 0.80 x 0.86 mm (rectangular field of view, 165 x 220 mm; image matrix, 205 x 256), and an acquisition time of 9 minutes 30 seconds were used.
Lesion Simulation
By applying the described multiecho spin-echo inversion-recovery MR sequence, images of the left centrum semiovale of the brain in the symptomatic patient with MS were obtained. The T1 relaxation, T2 relaxation, and at the x and y coordinates—T1x,y, T2x,y, and (x,y), respectively—for all pixels in the MS lesion were derived from the corresponding pixel values of the T1 relaxation, T2 relaxation, and maps, respectively.
Only pixels with T1 relaxation, T2 relaxation, and values 2 standard deviations or more above the average pixel values in the corresponding normal-appearing white matter (right centrum semiovale) were included in the MS lesion. This criterion was chosen on the basis of findings in a previous work of Melhem et al (8), who found that this threshold enables effective extraction of the lesion, at least at visual inspection. The extracted lesion had a 7-pixel diameter in each axial dimension. Then, by resampling the original lesion by using bicubic interpolation, we generated smaller lesions with diameters of 2, 3, or 4 pixels and placed them in the brain templates.
Brain Image Simulation
The generation of normal brain images, performed by two authors (L.P. and J.H.W., with less than 1 year and 7 years of experience, respectively), involved two steps: First, pixel-by-pixel T1 relaxation, T2 relaxation, and brain maps (256 x 256 matrix) were generated online by using computer software (release 6.2; Philips Medical Systems) and the mixed multi-echo spin-echo inversion-recovery image data sets. Second, images simulating FLAIR sequences were generated offline by using computer software (SUN enterprise 5500; Sun Microsystems, Mountain View, Calif) and T1 relaxation, T2 relaxation, and pixel values from the corresponding maps.
To develop the image-simulation software, we used Interactive Data Language (IDL; Research Systems, Boulder, Colo). This software applied an approximate solution to the Bloch equation by relating signal intensity (S) in terms of three parenchymal parameters—, T1 relaxation, and T2 relaxation at the x and y coordinates—and three imaging parameters—repetition time (TR), TE, and inversion time (TI)—at the x and y coordinates:
 | (1) |
Images of FLAIR contrast were created by applying this equation on a pixel-by-pixel basis. Parameters for the FLAIR sequence were selected on the basis of values typically used in clinical brain imaging—a repetition time of 11 000 msec and an inversion time of 2600 msec—whereas various TEs from 100 to 200 msec in 10-msec intervals were selected, for a total of 11 TEs.
A lesion was embedded into the brain template image by choosing the greater of the two signal intensities (lesion or brain) calculated by using Eq (1). At either axial level, 18 lesions were chosen so that the three lesion sizes (ie, six lesions each of sizes 2, 3, and 4 pixels) and three parenchymal locations (ie, six lesions each in cortical, deep white matter, and periventricular locations) were represented equally. For each of the 11 TEs used, these same 18 lesions were distributed among the 12 images to produce three images without lesions, four images with one lesion each, two images with two lesions each, two images with three lesions each, and one image with four lesions. In summary, 24 images containing 36 pseudorandomly placed lesions for each of the 11 TEs were generated to yield a total of 264 images. These 264 images were used to assess the performance of each reader in the detection of MS lesions (Figs 1, 2).
View larger version (99K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 2: Computer-generated FLAIR MR images of the infratentorial region of the brain obtained at various TEs. A, Normal image without MS Lesions. Images obtained at TEs of, B, 100 msec, and, C, 120 msec show the same three MS lesions (arrows) in cortical gray matter, in deep white matter, and periventricularly, distributed pseudorandomly. Images obtained at TEs of, D, 150 msec, E, 170 msec, and, F, 200 msec show the same three lesions. As in the supratentorial region, in the infratentorial region shown, lesion visibility decreases with increasing TE.
|
|
Observer Performance Testing
Seven board-certified neuroradiologists (including R.L.W., E.H.H., G.M., and E.R.M.) with various levels of experience (mean, 3.5 years; range, 1–11 years) independently evaluated the images, which were presented by using specialized testing software written in Interactive Data Language. In a 2-minute demo testing session conducted before the actual testing began, the reader was allowed to practice the point-and-click interface (described in upcoming text). None of the actual test images was included in the demo testing session. The reader was informed beforehand that each image could contain any number of lesions, including zero. No information about the maximal number of lesions was given. Each testing session was performed at the same desktop personal computer workstation (Dell Optiplex GX280; Dell, Round Rock, Tex) with the same 15-inch monitor (resolution, 1024 x 768 pixels) and uniform window width and center levels, which were chosen to match the parameters used in clinical practice.
The 264 images, each of which was expanded to a 512 x 512 matrix with use of linear interpolation, were randomly displayed one at a time so that supratentorial images were alternated with infratentorial images. When the reader perceived a possible lesion, he or she located it by using a simple point-and-click mouse interface. A drop-down menu then appeared and allowed the reader to assign a rating by using the following scale: a rating of 4 meant definite lesion, 3 meant probably lesion, 2 meant possible lesion, and 1 meant cannot exclude lesion. The software program recorded both the location and the rating, which constituted the reader's response. If desired, the reader was able to delete a response by using a different button click. The software allowed this function, for example, in cases in which the reader mistakenly clicked on an unintended location or simply reconsidered his or her prior response. The reader was instructed to designate and rate all detected lesions on each of the 264 images.
A maximum time limit of 30 seconds per image was enforced; after this time, the software would automatically erase the image from the screen. At the 25-second mark, a visual signal indicated that only 5 seconds remained to view the given image. The readers were not given any specific criteria for identifying possible MS lesions. It was left up to them to identify all possible MS lesions on each image according to their personal judgment, training, and experience. If satisfied that all possible lesions were selected, the reader could proceed to the next image before the end of the 30-second period by clicking on the button marked "Next."
Reader Response Scoring
A true-positive response score was assigned if the reader's response was within 5 pixels at both the x and the y coordinates of the lesion center. The reader did not need to click on the exact center to be assigned a true-positive response score. Each true-positive response was then recorded as a "hit" of its corresponding lesion rating of between 1 and 4. Similarly, a false-negative response score was assigned if the reader did not see the lesion or judged the parenchyma in question to be normal. As a result, a hit of lesion rating 0 was recorded. In this way, each lesion was assigned a "hit" score of between 0 and 4. Any response that did not correspond to an observer response of within 5 pixels at both the x and the y coordinates in the center of a true lesion was deemed to be false-positive. For each image, a false-positive image score that equaled the highest confidence score of the false-positive responses within that image was recorded. A false-positive image score of 0 corresponded to an image with no false-positive lesions—that is, a true-negative image.
Statistical Analyses
Two authors (L.P., J.H.W.) used a free-response receiver operating characteristic (FROC) method and a jackknife FROC software program (9) to analyze the data. This program computes a figure of merit statistic (
) that corresponds to a trapezoidal approximation of an area under the alternative FROC curve. In essence, the statistic serves as a quantitative measure of observer performance, because it also equals the probability of a signal intensity rating (ie, a "hit" score for a lesion) exceeding a noise rating (ie, a false-positive score for an image).
To compare the observers' performances in detecting lesions on FLAIR images at various TEs, confidence intervals for the statistic had to be estimated. The jackknife FROC program uses a jackknifing technique whereby pseudovalues are computed. Each time a case is deleted from the above sum of cases, a value is calculated for the remaining cases. More explicitly, pseudovalues (PVjik) for the reader (j), modality (i), and case (k) were calculated by using the following equation:
 | (2) |
where ij is the figure of merit for the ith modality and the jth reader when all cases are used in the calculation, ij(k) is the figure of merit for the ith modality and the jth reader when case k is deleted, and NT is the total number of cases (10).
These pseudovalues, whose variance reflected the confidence intervals for , were used to compute the F statistic needed to determine whether a difference in values between two different TEs was statistically significant. The sample variance of the computed pseudovalues represented an estimate of the variance in the performance measure due to case variability. P < .05 was chosen as an indication of significance. The computed values, expressed as percentages, were averaged to yield a pooled score that reflected the overall performance. The total variance in this performance measure, (
)2, could then be determined by summing the contributions due to interreader variability, (read)2, and intercase variability, (PV)2.
|
RESULTS
|
|---|
Each neuroradiologist independently evaluated the same set of 264 images, which consisted of 11 subsets of 24 images with 36 MS lesions distributed pseudorandomly. The total time allotted for each reader for a testing session was 132 minutes (264 images times 30 seconds each), but each observer completed the test within 2 hours. The individual observer performance, or value, calculated for each subset of images varied substantially at both the supratentorial and the infratentorial levels: from 41% to 100% for both regions (Tables 1, 2). The ranges of averaged individual performance values calculated for all TEs were smaller but still similar supra- and infratentorially—27% and 29%, respectively (Tables 1, 2)—whereas the mean of averaged performance values for the supratentorial region was about 10% higher than the respective mean value for the infratentorial region.
View this table:
[in this window]
[in a new window]
|
Table 1. Individual Performance Values for Seven Neuroradiologists in the Detection of MS Lesions Distributed Supratentorially on FLAIR Images
|
|
View this table:
[in this window]
[in a new window]
|
Table 2. Individual Performance Values for Seven Neuroradiologists in the Detection of MS Lesions Distributed Infratentorially on FLAIR Images
|
|
The average overall observer performance value calculated for all neuroradiologists and across both regions differed significantly among the TE values (P < .05, repeated-measures analysis of variance) and was highest (90%) at the 100-msec TE. Observer performance decreased almost linearly (r = –0.965, P < .05) (Fig 3) with increasing TE. Overall performance was highest at the 100-msec TE supratentorially and infratentorially: 93% and 87%, respectively (Tables 1, 2). The decreasing performance with increasing TE, however, was even more apparent infratentorially (r = –0.94 compared with r = –0.97 supratentorially) (Tables 1, 2). Despite the equal numbers of images and lesions available for assessment for each TE, the variability in observer performance was the lowest and the precision of the observer performance measurement, as determined by using means ± 1.96 standard errors of the mean, was the highest at a TE of 100 msec, both supra- and infratentorially (Tables 1, 2).
View larger version (18K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 3: Plot shows an almost linear relationship between TE and mean overall (for both supratentorial and infratentorial regions) observer performance in the detection of MS lesions on FLAIR MR images of the brain. Error bars represent standard errors of the mean.
|
|
|
DISCUSSION
|
|---|
The results of this study show that the TE selection has a significant effect on observer performance in the detection of artificial MS lesions on FLAIR MR images. Among the 11 TEs studied, from 100 to 200 msec, the shortest one yielded the best performance both supra- and infratentorially. The performances of the most experienced neuroradiologists were almost perfect at 100 msec. Observer performance at TEs shorter than 100 msec was not assessed because according to the findings in a previous study, which showed optimal contrast between MS lesions and white matter at TEs of 130–160 msec, we believed that optimal observer performance would be achieved at a TE of between 100 and 200 msec (7). We can find no consensus in the literature, however, regarding the optimal TE for the detection of MS lesions with FLAIR imaging.
Previous study investigators (2–6) have used different TEs from 119 to 150 msec. However, their selection of a TE was empirical and possibly based on personal preference. Our study results provide a scientific basis for selecting a TE. On the basis of our results, the optimal TE appears to be shorter than those used in these previously published studies. This theory implies that the TEs routinely used in clinical practice to examine patients with MS at FLAIR imaging are probably too long.
Our study findings indicate that observer performance in the detection of MS lesions decreases almost linearly as TEs increase from 100 to 200 msec, especially in the infratentorial region. This substantial decrease in performance may explain why the accuracy of FLAIR MR imaging in the detection of MS lesions in the posterior fossa has been lower than that of T2-weighted MR imaging (3,4,6,11). In our opinion, the TEs that we used are well above the optimal values used in virtually all published studies on the use of FLAIR imaging in patients with infratentorial MS lesions (3,4,6,11). Optimization of the TE can improve the detectability of MS lesions at FLAIR imaging, which may surpass the detectability at T2-weighted imaging—the current preferred technique for detecting MS lesions in the posterior fossa. Further work is warranted to test the hypothesis of the superiority of FLAIR imaging over T2-weighted imaging in the detection of MS lesions in the posterior fossa, however, since the lesion visibility on FLAIR images is compromised not only by parenchymal contrast but also by artifacts.
In our study, observer performance was quantified by using a recently developed jackknife FROC paradigm and an associated analysis method, which allow the observer to report multiple abnormalities within each imaging examination and involve the use of the location of the reported abnormalities to improve the measurement. This method (10), in contrast to the earlier FROC method, does not rely on an assumption of the independency of responses made regarding the same diagnostic examination. Furthermore, the current analysis method has more statistical power than the previous FROC and multireader multicase Dorfman-Berbaum-Metz methods.
The modeling of observer performance studies has been a problem of previously published work (12). This is because investigators most often are concerned with not only the detection of an abnormality but also the location of the lesion in specific image regions. The jackknife FROC model is based on current thinking about how observers scan images for lesions. In clinical interpretations, observers do not know the locations of any lesions that may be present on an image and differ in their ability to search the image and find the lesions. The jackknife FROC method, by yielding a single measure of observer performance ( statistic), enabled us to quantify and compare performances between observers and across all locations.
By using the simulator, we tried to replicate an actual clinical scenario as closely as possible. Each observer evaluated all 264 images. This number of images was selected to ensure that each observer did not spend more than 2 hours participating in the study. We thought that 2 hours was a reasonable amount of time for each observer to read the images without becoming fatigued. We could have included more images by splitting the observer testing session into two or more parts. However, doing this would have introduced an additional variance component to the observer performance measurements.
A period of about 30 seconds per image does not simulate the interpretation time in typical clinical settings, however, because radiologists usually look at each image for less than 10 seconds. The longer time that the observers were given per image in this study may have led to an increased number of false-positive findings. However, since the observers were allowed the same amount of time per image, regardless of the TE, the possible increase in the number of false-positive findings would have occurred at each TE.
Although the same number of images was used for each TE, interobserver variability was the lowest at TEs of 100 and 110 msec and the highest at TEs of 190 and 200 msec. Consequently, the precision of the estimation of observer performance was highest at the shortest TE and lowest at the longest TE. This result implies that the desired consistency of the observers in detecting lesions, regardless of their experience, was best at a TE of 100 msec. This is yet more proof that a TE of 100 msec may be the optimal value for FLAIR MR imaging. To estimate true observer performance with greater precision, a larger number of images per TE are required.
Our study had other limitations, which were related mainly to the use of the MS lesion simulator. First, only one signal intensity value for all artificial MS lesions was used, and this could have inflated the observers' sensitivities compared with the sensitivities of observers in actual clinical settings. This limitation was counterbalanced by the use of only the smallest—and therefore the most difficult to detect—lesions. Another factor, which could have inflated the observers' accuracy, was the use of only one template image each supratentorially and infratentorially. The observers repeatedly viewed the same template image at each TE. This factor could have led to a recalling effect, because an observer could have memorized the presence of the lesion in a particular location on one image and its absence at that location on another image. All images obtained with different TEs, however, were displayed pseudorandomly to equalize the influence of the recalling effect—if any—on observer performance in the detection of lesions at particular TEs.
In addition, we used only one extracted white matter lesion for all image simulations. We did not digitally extract a gray matter lesion. Therefore, the simulated images of lesions in the gray matter may not have accurately reflected what an actual gray matter lesion looks like. In our opinion, however, our results are valid, because any of the potential limitations just described would have affected the observer performances determined at all TEs to the same degree.
In conclusion, by using simulated FLAIR brain MR images containing artificial MS lesions, we found that both observer performance and observer consistency in the detection of MS lesions were best at a TE of 100 msec, both supratentorially and infratentorially. Thus, our results may be helpful in subsequent clinical trials.
|
ADVANCE IN KNOWLEDGE
|
|---|
- FLAIR parameters are optimized in the detection of MS lesions in infratentorial and supratentorial brain regions.
|
FOOTNOTES
|
|---|
Abbreviations: FLAIR = fluid-attenuated inversion recovery • FROC = free-response receiver operating characteristic • MS = multiple sclerosis • TE = echo time
Author contributions: Guarantors of integrity of entire study, L.P., J.H.W., J.K., E.R.M.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, L.P., J.H.W., A.F.J., J.K., E.R.M.; experimental studies, L.P., J.H.W., R.L.W., E.H.H., G.M., J.K., E.R.M.; statistical analysis, L.P., J.H.W., A.F.J., J.K., E.R.M.; and manuscript editing, all authors
Authors stated no financial relationship to disclose.
|
References
|
|---|
- Miller DH, Filippi M, Fazekas F, et al. Role of magnetic resonance imaging within diagnostic criteria for multiple sclerosis. Ann Neurol 2004;56:273–278.[CrossRef][Medline]
- Herskovits EH, Itoh R, Melhem ER. Accuracy for detection of simulated lesions: comparison of fluid-attenuated inversion-recovery, proton density–weighted, and T2-weighted synthetic brain MR imaging. AJR Am J Roentgenol 2001;176:1313–1318.[Abstract/Free Full Text]
- Bastianello S, Bozzao A, Paolillo A, et al. Fast spin-echo and fast fluid-attenuated inversion-recovery versus conventional spin-echo sequences for MR quantification of multiple sclerosis lesions. AJNR Am J Neuroradiol 1997;18:699–704.[Abstract]
- Yousry TA, Filippi M, Becker C, Horsfield MA, Voltz R. Comparison of MR pulse sequences in the detection of multiple sclerosis lesions. AJNR Am J Neuroradiol 1997;18:959–963.[Abstract]
- Moriarty DM, Blackshaw AJ, Talbot PR, et al. Memory dysfunction in multiple sclerosis corresponds to juxtacortical lesion load on fast fluid-attenuated inversion-recovery MR images. AJNR Am J Neuroradiol 1999;20:1956–1962.[Abstract/Free Full Text]
- Rovaris M, Comi G, Rocca MA, et al. Relevance of hypointense lesions on fast fluid-attenuated inversion recovery MR images as a marker of disease severity in cases of multiple sclerosis. AJNR Am J Neuroradiol 1999;20:813–820.[Abstract/Free Full Text]
- Rydberg JN, Riederer SJ, Rydberg CH, Jack CR. Contrast optimization of fluid-attenuated inversion recovery (FLAIR) imaging. Magn Reson Med 1995;34:868–877.[Medline]
- Melhem ER, Herskovits EH, Karli-Oguz K, et al. Defining thresholds for changes in size of simulated T2-hyperintense brain lesions on the basis of qualitative comparisons. AJR Am J Roentgenol 2003;180:65–69.[Abstract/Free Full Text]
- Chakraborty DP, Winter LHL. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology 1990;174:873–881.[Abstract/Free Full Text]
- Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys 2004;31:2313–2330.[CrossRef][Medline]
- Okuda T, Korogi Y, Shigematsu Y, et al. Brain lesions: when should fluid-attenuated inversion-recovery sequences be used in MR evaluation? Radiology 1999;212:793–798.
- Dorfman DD, Berbaum KS, Metz CE. ROC characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992;27:723–731.[CrossRef][Medline]
TOP
ABSTRACT
INTRODUCTION
