| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Gastrointestinal Imaging |
1 From the Department of Specialist X-Ray (S.A.T., S.H.) and Medical Statistics Unit (S.C.C.), University College Hospital, 235 Euston Rd, 2F Podium, London NW1 2BU, England; Department of Radiology, Stedelijk Ziekenhuis, Roeselare, Belgium (P.L.); Department of Radiology, Mallinckrodt Institute of Radiology, St Louis, Mo (E.G.M.); Abdominal Imaging Section, Duke University Medical Center, Durham, NC (E.K.P., C.M.M.); Department of Radiology, San Francisco VA Medical Center, University of California, San Francisco, Calif (J.Y., R.A.); Department of Radiology, Mayo Clinic College of Medicine, Rochester, Minn (J.M.B.); Department of Intestinal Imaging, St Mark's Hospital, Harrow, United Kingdom (A.G.); and Department of Radiology, University of Wisconsin Medical School, Madison, Wis (D.H.K.). From the 2006 RSNA Annual Meeting. Received January 29, 2007; revision requested March 27; revision received May 9; final version accepted June 11. This work was supported in part by the U.K. Department of Health's NIHR Biomedical Research Centres. Address correspondence to S.A.T. (e-mail: csytaylor{at}yahoo.co.uk).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
Materials and Methods: Ethical permission, HIPAA compliance (for U.S. institutions), and patient consent were obtained from all institutions for use of CT colonography data sets in this study. Ten radiologists each read 25 CT colonography data sets (12 men, 13 women; mean age, 61 years) containing 69 polyps (28 were 1–5 mm, 41 were 
6 mm) by using workstations integrated with CAD software. Reading was randomized to either "second read" CAD (applied only after initial unassisted assessment) or "concurrent read" CAD (applied at the start of assessment). Data sets were reread 6 weeks later by using the opposing paradigm. Polyp sensitivity and reading times were compared by using multilevel logistic and linear regression, respectively. Receiver operating characteristic (ROC) curves were generated.
Results: Compared with the unassisted read, odds of improved polyp (6 mm) detection were 1.5 (95% confidence interval [CI]: 1.0, 2.2) and 1.3 (95% CI: 0.9, 1.9) by using CAD as second and concurrent reader, respectively. Detection odds by using CAD concurrently were 0.87 (95% CI: 0.59, 1.3) and 0.76 (95% CI: 0.57, 1.01) those of second read CAD, excluding and including polyps 1–5 mm, respectively. The concurrent read took 2.9 minutes (95% CI: –3.8, –1.9) less than did second read. The mean areas under the ROC curve (95% CI) for the unassisted read, second read CAD, and concurrent read CAD were 0.83 (95% CI: 0.78, 0.87), 0.86 (95% CI: 0.82, 0.90), and 0.88 (95% CI: 0.83, 0.92), respectively.
Conclusion: CAD is more time efficient when used concurrently than when used as a second reader, with similar sensitivity for polyps 6 mm or larger. However, use of second read CAD maximizes sensitivity, particularly for smaller lesions.
© RSNA, 2007
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
The classic CAD implementation, "second reader" CAD, is applied only after the observer has completed a full, unaided assessment (8). In this paradigm, CAD acts as a safety net for abnormalities missed by the unassisted reader. However, such implementation increases interpretation time. A potentially more time efficient paradigm is "concurrent read" CAD, which applies CAD at the start of the assessment. Although an intuitively attractive proposition, there is some evidence from breast (9) and pulmonary (10) literature that concurrent application of CAD reduces observer vigilance, reducing sensitivity. It is unclear whether this phenomenon applies to CT colonography.
Thus, the purpose of our study was to prospectively compare the diagnostic performance and time efficiency of both second read and concurrent read CAD paradigms for retrospectively obtained CT colonography data sets, by using consensus reading (S.A.T., S.H., with experience of 600 and 800 endoscopically validated CT colonography datasets, respectively; and a nonauthor) of colonoscopic findings as a reference standard.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
Ethical permission, HIPAA compliance (for U.S. institutions), and patient consent were obtained from all donor institutions (Hull Royal Infirmary, Hull, England; St Marks Hospital, Harrow, England; and Beth Israel Deaconess, Boston, Mass) for use of the CT colonography data sets in this study.
Reader Selection
Five radiologists (P.L., E.G.M., E.K.P., J.Y., and a nonauthor, each with experience in 300–800 CT colonography examinations) agreed to take part in the study. Each radiologist selected one less-experienced institutional colleague (A.G., D.H.K., J.M.B., R.A., C.M.M.) who had undergone formal training in CT colonography interpretation (either attendance at a formal 2-day training course or completion of a teaching file of 50 endoscopically validated data sets) and interpreted CT colonographic images unassisted in everyday clinical practice but lacked the experience of a large volume of verified data sets (range, 65–80). In total, 10 observers participated in the study.
Selection of CT Colonography Data sets
On the basis of our previous data (3), we calculated the interclass coefficient (an estimate of agreement between multiple observers) as 0.4–0.5 for CAD-assisted interpretation and the design effect as 4.6.
A power calculation (
= .05, 80% power) performed on the basis of these estimates from 10 readers suggested that 40 polyps 6 mm or larger were required to detect at least a 15% difference in polyp detection between the two reader paradigms described below.
On the basis of this calculation, 25 CT colonography data sets (five normal) containing 41 polyps (6 mm) were chosen at random from a research database that was collated from three donor institutions with research programs comparing CT colonography with same-day endoscopy performed with segment unblinding (Fig 1). All patient data sets in the research database were obtained from institutions with the appropriate ethical permission for data sharing and were Health Insurance Portability and Accountability Act–compliant for U.S. institutions.
|
Reference Standard
The 25 studies were loaded on a workstation equipped with CT colonography viewing software (Medicolon 1.2; Medicsight) and read in consensus by three radiologists (S.A.T., S.H., and a nonauthor, experienced in 400–800 endoscopically validated CT colonography studies each) with full knowledge of colonoscopic findings. Readers indicated the location of polyps by noting image section numbers and by drawing an outline around the polyp circumference with a freehand-drawing software tool on the appropriate two-dimensional (2D) transverse image. Polyps were matched to the colonoscopic reference by using segment location (within same or adjacent segment) and size (CT measurement within 50% of the colonoscopic size).
Study Workstation and CAD Integration
Data sets were then loaded onto 10 identical workstations (Vitrea 2; Vital Images) integrated with software (ColonCAR 1.2; Medicsight). The commercially available CAD system used for our study has been described elsewhere (7,11). In brief, the software segments the colon seen in the CT data set and determines the inherent sphericity of all raised objects. Detected objects with a sphericity above a predetermined threshold value are then prompted to the observer with small red dots superimposed over the region of interest on 2D transverse and three-dimensional (3D) endoluminal views (Fig 2), or by a yellow triangle when the polyp candidate is hidden behind a fold during 3D endoluminal analysis (Fig 3). The user can influence the threshold for prompted polyps by using slider bars with an arbitrary scale between 0.50 (most sensitive) and 1.00 (least sensitive). The default setting was 0.75 (12).
|
|
|
Each participant independently read the 25 studies on two occasions and each reading was separated by a period of 6 weeks to minimize recall bias. Reading sessions were scheduled to coincide with an international radiologic conference attended by all study participants, and workstations were available for a minimum of 4 days. Readers were encouraged to read the studies in small blocks, mirroring normal clinical practice, rather than attempting all 25 at one sitting. During the first reading session, the data sets were analyzed by using one of the two paradigms (second read or concurrent read CAD—see below) allocated at random on a per-case basis for each reader.
Six weeks later, case ordering was randomized again, and studies were reinterpreted with the opposing CAD paradigm. Readers were fully blinded to the prevalence of abnormality and were not told they would reread the same 25 studies during the second session. Readers were instructed to read the study as they would in normal clinical practice and were not told to ignore polyps below any particular size threshold. However, the polyp size threshold used by each reader was recorded after completion of reading of the studies.
Reading Paradigms
For both reading paradigms, readers were free to use the full functionality of the workstation (ie, 2D transverse images, multiplanar reconstruction, 3D cube view, and full endoluminal flythrough). Data sets were preloaded on each workstation, allowing preprocessing with the CAD algorithm so that CAD marks appeared instantaneously when activated by the reader. Readers could not alter the size of the CAD mark, but the total number of CAD marks for checking was listed by the software.
Second reader CAD.—Readers were instructed to first analyze the entire case without CAD (unassisted read), recording interpretation time (defined as time taken to read the data set once opened on the workstation), and to document each perceived abnormality, noting the colonic segment, 2D transverse section numbers, lesion size (in millimeters), and diagnostic confidence (from 1 [least confident] to 100 [most confident]). Readers were also instructed to make screensaves of all detected lesions and include these in a final printed report (the time to produce the printed report was not included in the data set interpretation time).
Once this analysis was complete, readers applied CAD and then reviewed the case again. Readers could jump straight to sequential CAD marks by pressing the keyboard space bar if they wished. Readers documented any additional findings after applying CAD, exactly as before, and the time taken for additional CAD-assisted review was recorded. Readers were also permitted to disregard findings on the unassisted read after the application of CAD.
Concurrent reader CAD.—Readers were instructed to apply the CAD algorithm immediately prior to their review so that interpretation was performed once with superimposed CAD marks from the outset. Readers were able to turn the marks on and off with a single mouse click if they wished. Documentation of perceived abnormalities, diagnostic confidence, and reading time was performed as described above.
Case Marking
A nonobserver trained in CT colonography marked reports on a per-polyp basis for each reader by comparing the annotated consensus reference standard for each case with the report made by the reader. A reader detection was considered a true-positive if the section numbers and colon segment matched the reference standard. Any potential discrepancy (eg, incorrect colon segment or section numbers) was resolved by recalling the reader report and comparing the polyp snapshot with that of the reference standard. Correct observations were allowed if reader measurement was within 50% of the true colonoscopic size.
Statistical Analysis
All statistical analyses were performed by using software (Stata, version 9, Stata, College Station, Tex; and MLwiN, version 2.02, University of Bristol, Bristol, England). Significance was assigned for a P value of less than .05 when appropriate. Polyps were sorted according to size: small (1–5 mm), medium (6–9 mm), and large (10 mm). Polyp sensitivity (overall, and only for polyps 6 mm) was compared with CAD reader paradigms and with the unassisted read by using multilevel logistic regression.
Polyps and readers were considered as random cross-classifications, with polyp diameter, reading method, and expertise considered as fixed factors. The effect of polyp size, reading paradigm, and reader experience (and any interaction) on the odds of polyp detection was assessed by using Wald test or joint Wald test as appropriate. Tests of interaction assess whether there is any important effect modification, for example, whether the effect of one factor (such as reader paradigm) changes for differing levels of other factors (such as reading experience).
Data were presented as odds ratios (ORs) and 95% confidence limits. The OR is interpreted as the ratio of the odds of detection for one group or circumstance (eg, experienced readers) compared with the odds for another (eg, trained readers). Reading times between reader paradigms were compared by using a multilevel linear model in which patients and readers were considered as random cross-classifications.
Method and expertise were included and assessed as above. Observer false-positives (overall, and only for polyps 6 mm) were assumed to follow a Poisson distribution and were analyzed by using a multilevel log-linear model, with data sets and readers considered as random cross-classifications and reading paradigm and reader expertise considered as fixed factors. Data were expressed as rate ratios.
Receiver operating characteristic (ROC) curves were generated for each reader according to their ability to correctly classify each data set as normal or abnormal based on CT Colonography Reporting and Data System guidelines (13) (ie, only a polyp 6 mm or larger was considered abnormal). The highest confidence level assigned to each data set was noted and treated as a continuous variable against the reference classification (normal or abnormal). ROC curves were compared by using mixed-effects linear regression, with the area under the ROC curve (AUC) defined as a dependent variable, the reader as a random effect, and the reading method and reader experience as fixed factors. The importance of fixed factors was defined by likelihood ratios.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
10 mm [large]). At the default sphericity of 0.75, CAD helped detect 37 (90%) of 41 polyps 6 mm or larger (11 [39%] of 28 polyps 1–5 mm and 18 [95%] of 19 polyps 10 mm), with a median of nine (supine) and 10 (prone) false-positive prompts per data set. Of 250 readings, readers used a primary 2D approach for 226 (90.4%) and 221 (88.4%) of the second read and concurrent read paradigms, respectively. Of the 10 readers, four stated that, in general, they ignored polyps 5 mm or less during the study. All readers analyzed the 25 data sets in no less than three separate sittings during both main reading sessions.
Polyp Sensitivity
All two-way interactions and the three-way interaction between polyp diameter, expertise, and reading method were not significant and therefore not included in the final model.
Polyps 6 mm or larger.—Observer experience influenced polyp detection, with the odds of detection by experienced readers 2.5 (95% confidence limits: 1.1, 5.6) times greater than those of less experienced readers (Table 1, Fig 4).
|
|
6 mm) detection by using CAD as a second reader (OR, 1.5 [95% confidence limits: 1.0, 2.2]) were greater than were those using CAD concurrently (OR, 1.3 [95% confidence limits: 0.9, 1.9]). Across all readers, detection of polyps 6 mm or larger was 77% (95% CI: 74, 79), 83% (95% CI: 81, 84), and 81% (95% CI: 79, 83) for unassisted read, second reader CAD, and concurrent read CAD, respectively. All polyps.—When small polyps (1–5 mm) were included in the model, the effect of reading method on polyp detection approached, but did not reach, significance (P = .06). However, there was some evidence of improved sensitivity by using a second read paradigm compared with the concurrent read (detection odds with a concurrent read was 0.76 [95% confidence limits: 0.57, 1.01] that of the second read paradigm). Overall, seven of 10 readers detected more polyps by using a second read paradigm than by using the concurrent read (Fig 5). Compared with the CAD unassisted read, the odds of polyp (any size) detection by using CAD as second reader (OR, 1.4 [95% confidence limits: 1.1, 1.8]) were again greater than were those using CAD concurrently (OR, 1.04 [95% confidence limits: 0.78, 1.39]).
|
6 mm, all identified with CAD) were identified during the CAD component of the second read paradigm by at least one reader who had originally missed the polyp on the unassisted read. Conversely, only one polyp (4 mm, not identified with CAD) identified on the unassisted read was subsequently incorrectly dismissed after the application of second read CAD.
Compared with the unassisted read, 33 polyps (10 were 1–5 mm, 23 were 6 mm) were additionally detected by at least one reader during the concurrent read paradigm, of which 30 (91%) were detected with CAD. However, 40 individual polyps (17 were 1–5 mm, 23 were 6 mm) were missed by at least one reader during the concurrent CAD read and were originally correctly identified on the unassisted read. Of these 40 polyps, 27 (68%) were detected with CAD and 13 (32%) were not.
False-Positive Detection
On average, readers reported at least one false-positive result for a polyp 6 mm or larger in 18.8%, 26.4%, and 27.2% of data sets by using unassisted read, second read CAD, and concurrent read CAD, respectively (Tables 2, 3; Fig 6).
|
|
|
Reading paradigm also significantly influenced the number of false-positives recorded (P = .008). Compared with the unassisted read, the rates of false-positive detections of any size were 1.5 (95% CI: 1.2, 1.9) and 1.3 (95% CI: 0.98, 1.6) when using the second and concurrent reading paradigms, respectively, and rate ratios of 1.6 (95% confidence limits: 1.1, 2.2) and 1.6 (95% confidence limits: 1.2, 2.3), respectively, when limited only to false-positive results 6 mm or larger. However, on a per-patient basis (Table 3), there was little difference between the paradigms.
ROC Analysis
The lowest reader confidence value included in the ROC analysis for abnormal and normal data sets was 10 (Fig 7).
|
Reading Times
Reading method (but not reader experience) was significantly related to reading time (P < .001). Mean reading times for unassisted read, second read CAD, and concurrent read CAD were 11 (95% CI: 9.2, 13), 15 (95% CI: 13, 17), and 12 (95% CI: 10, 14) minutes, respectively. The use of CAD as a second reader added 3.7 (95% CI: 2.7, 4.7) minutes to the unassisted read. The concurrent read paradigm took less time than did the second read paradigm (–2.9 minutes [95% CI: –3.8, –1.9]) and was 0.84 minute (95% CI: –0.13, 1.81) longer than the unassisted read.
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
We did not find any definite evidence of a difference between the two CAD reader paradigms in the detection of polyps 6 mm or larger. Interestingly, when small (1–5 mm) polyps were included in the model, the differences between the two CAD paradigms became more marked with odds of detection 30% higher, on average, when using CAD as a second read. We deliberately powered our study only with polyps 6 mm or larger, which the radiology community has considered clinically important (15). The importance of polyps 5 mm or smaller remains controversial within the gastroenterology community, not least because the risks of polypectomy may outweigh the potential risk of leaving small polyps in situ (16).
However, it is known that multiple small polyps increase the risk of colorectal cancer, so detection and removal are still advocated (17). We thus analyzed our data with and without these diminutive lesions. We did not want to bias our readers by instructing them to ignore small polyps, instead preferring them to read as they would during normal practice. Of the 10 readers, four stated that, in general, they ignored polyps 5 mm or smaller in our study, although we included all 10 in the analysis to capture all available data.
It is likely that individual reader threshold levels for reporting or ignoring small polyps remained constant throughout our study. Given this, the second read paradigm held an advantage over concurrent read if small polyps were considered important in any way. The CAD used in our study was trained to detect polyps 4 mm or larger (hence its inferior performance for diminutive lesions), such that data for these small lesions should be viewed with some caution. Nevertheless, it is interesting that CAD (particularly second read) improved reader performance, even for diminutive lesions, despite the low CAD detection.
It is interesting to speculate why some polyps detected by readers without the benefit of CAD are ignored during a concurrent read. It seems reasonable to expect readers to rush analysis of the unprompted colon image during the concurrent read, instead focusing on each CAD mark. We did find some evidence supporting this; CAD did not prompt 32% of the polyps missed by at least one reader that were correctly identified on the unassisted read. By definition, 68% of polyps missed were correctly marked by CAD. It is uncertain why readers ignored more CAD-prompted polyps when reading concurrently but interestingly, Zheng et al (9) demonstrated that reduced mammographic sensitivity for breast cancer by using concurrent reading was worse when the CAD false-positive rate was highest, suggesting that multiple prompts result in readers placing less weight on those which are correct.
Our data suggest a similar phenomenon exists for CT colonography with CAD, especially for small polyps. The concurrent read was significantly more time efficient than the second read, which has implications for work flow. Arguably, time efficiency far outweighs the reduced sensitivity for diminutive polyps.
Because we wished to limit our tested intervention to the effect of CAD only, we allowed readers to analyze the data sets as they would during their usual practice, without dictating the reading paradigm. Most readers used a primary 2D approach, rather than 3D endoluminal analysis. Both the specific workstation used and the use of nontagged data sets may have influenced readings and it is interesting to speculate whether our results would have been different if a predominantly 3D endoluminal approach had been used. Future study is required to investigate the interaction between CAD and the reading method.
There is already good evidence that experience in CT colonography confers a performance advantage (18,19). Although the trained readers in our study performed well, experienced readers benefited proportionately less from CAD. Although CAD correctly prompted 90% of polyps 6 mm or larger, reader performance, although good, was inferior to this. A recent study (11) found that average detection by 10 readers (untrained in CT colonography) of polyps measuring 10 mm or larger was just 51%, despite CAD correctly prompting 89.5% of such lesions. Our study again emphasizes that excellent CAD performance alone does not translate into equivalent reader performance, underlining the complex interaction between CAD and observers.
Although the reader false-positive rate was relatively low overall, both CAD paradigms (particularly second read CAD) produced more reader false-positives than when unassisted, especially for experienced readers. However, our ROC analysis suggests that although CAD reduced specificity, this was disproportionately offset by increased sensitivity; AUCs were larger (by a small amount) than for the unassisted read.
Our study had limitations. We used a polyp-enriched data set to achieve adequate statistical power. Furthermore, it could be argued that a per-patient rather than a per-polyp analysis is more clinically relevant, and with only five normal data sets of 25, our per-patient ROC analysis was restricted. However, mimicking the low prevalence of abnormality found in asymptomatic practice would have meant an unfeasibly large data set, and our main aim was to specifically investigate the interaction between the reading paradigm used for CAD and the human observer, rather than investigate the clinical impact of CAD in specific patient populations. To our knowledge, our study is the first to address this issue.
We did not use data sets with oral contrast agents because not all readers had experience with such data sets, although this may have adversely influenced specificity. While we cannot exclude recall bias, the CAD paradigm used for each case was randomized between the two reader sessions, such that any recall bias would affect both paradigms equally. We attempted to mirror routine clinical practice by asking readers to review data sets over several days at two international conferences, although in reality, we accept that such conditions are somewhat removed from the normal hospital environment.
We did not stratify the polyps in our data sets according to histologic categorization. All lesions were visible at CT colonography and it seems unlikely that the results would differ significantly between adenomatous and hyperplastic polyp subtypes.
Finally, it could be argued that the CAD marks in the workstation integration used in our study are rather subtle, and we have no real way of being sure they were seen by the readers. The issues surrounding the optimal integration between CAD and workstation are complex (20) and the preferred CAD mark may differ between radiologists. Therefore, we cannot fully extrapolate our results to other CAD solutions.
In conclusion, use of CAD as a concurrent reader during CT colonography is more time efficient than use as a second reader, and similar sensitivity is achieved for polyps 6 mm or larger. However, use of CAD as a second reader maximizes sensitivity, particularly for smaller lesions.
|
ADVANCES IN KNOWLEDGE |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
|
FOOTNOTES |
|---|
Abbreviations: AUC = area under the ROC curve • CAD = computer-aided detection • CI = confidence interval • OR = odds ratio • ROC = receiver operating characteristic • 3D = three-dimensional • 2D = two-dimensional
See Materials and Methods section for pertinent disclosures.
Author contributions: Guarantor of integrity of entire study, S.A.T.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approved, all authors; literature research, S.A.T.; statistical analysis, S.C.C.; and manuscript editing, all authors
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSION ADVANCES IN KNOWLEDGE References |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
