Ovarian Cancer: Comparison of Observer Performance for Four Methods of Interpreting CT Scans1
Patrick J. Fultz, MD,
Christina V. Jacobs, MD 2,
W. Jackson Hall, PhD,
Ronald Gottlieb, MD,
Deborah Rubens, MD,
Saara M. S. Totterman, MD, PhD,
Steven Meyers, MD, PhD,
Cynthia Angel, MD,
Giuseppe Del Priore, MD, MPH 3,
David P. Warshal, MD 4,
Kelly H. Zou, PhD 5 and
David E. Shapiro, PhD 6
1 From the Depts of Radiology (P.J.F., C.V.J., R.G., D.R., S.M.S.T., S.M.), Obstetrics and Gynecology (C.A., G.D.P., D.P.W.), and Biostatistics (W.J.H., K.H.Z., D.E.S.), University of Rochester Medical Center, 601 Elmwood Ave, Rochester, NY 14642. Received May 7, 1997; revision requested May 27; final revision received Sep 2, 1998; accepted Feb 12, 1999. Supported in part by a grant from Innovations in Patient Care Program, Univ of Rochester Medical Center. Address reprint requests to P.J.F.

View larger version (29K):
[in a new window]
|
Figure 1. Diagram shows the format for the review of images from CT examinations and the reader (observer) assignments. For reading sessions 1-3, the 147 examinations were divided into three groups of examinations: group 1 (G1), 49 examinations; group 2 (G2), 49 examinations; and group 3 (G3), 49 examinations.
|
|

View larger version (32K):
[in a new window]
|
Figure 2. Abdominal-pelvic CT checklist. Modified from Kinard et al (10). GI = gastrointestinal, IVC = inferior vena cava.
|
|

View larger version (24K):
[in a new window]
|
Figure 3a. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (22K):
[in a new window]
|
Figure 3b. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (24K):
[in a new window]
|
Figure 3c. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (22K):
[in a new window]
|
Figure 3d. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (25K):
[in a new window]
|
Figure 3e. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (23K):
[in a new window]
|
Figure 3f. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (24K):
[in a new window]
|
Figure 3g. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (24K):
[in a new window]
|
Figure 3h. ROC curves and areas under the curves (AUC) for overall patient CT study evaluation (abnormal vs normal) for each observer in comparisons of single observer with single observer plus checklist and of single observer with paired observers. (a, b) Observer 1. (c, d) Observer 2. (e, f) Observer 3. (g, h) Observer 4. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (18K):
[in a new window]
|
Figure 4. ROC curves and areas under the curve (AUC) for categorizing abnormal and normal patient CT studies for the three interpretative methods. These curves were constructed by pooling all readings for each method. FPR = false-positive rate, TPR = true-positive rate.
|
|

View larger version (176K):
[in a new window]
|
Figure 5. False-negative diagnosis. CT image shows a tumor implant (arrow) at the psoas margin that was unrecognized at six of 10 reviews.
|
|

View larger version (171K):
[in a new window]
|
Figure 6. False-positive diagnosis. Pelvic CT image shows a mildly asymmetric vaginal cuff (arrow), which was mistaken for a tumor implant at three of 10 reviews. A vaginal tampon (arrowhead) is present on the right.
|
|
Copyright © 1999 by the Radiological Society of North America.