Published online before print September 11, 2007, 10.1148/radiol.2451061798
(Radiology 2007;245:411-415.)
© RSNA, 2007
A True Screening Environment for Review of Interval Breast Cancers: Pilot Study to Reduce Bias1
Paula B. Gordon, MD,
Marilyn J. Borugian, PhD, and
Linda J. Warren Burhenne, MD
1 From the Screening Mammography Program of British Columbia, British Columbia Cancer Agency, 686 W Broadway, 8th Floor, Vancouver, BC, Canada V5Z 1G1. From the 2006 RSNA Annual Meeting. Received October 18, 2006; revision requested December 19; revision received January 12, 2007; final version accepted February 12.
Address correspondence to P.B.G.
 |
ABSTRACT
|
|---|
Purpose: To retrospectively assess the feasibility of an uninformed review process to evaluate interval breast cancers and to compare the number of false-negative cancers detected at uninformed review with the number detected at standard informed review.
Materials and Methods: Institutional review board approval was obtained for this retrospective study, and informed consent was waived. Mammograms showing interval cancer were included in the daily work of radiologists in a high-volume screening center. Each of three experienced radiologists read studies in the normal screening environment, without knowledge that identifiers had been changed to conceal the fact that studies were not current (ie, uninformed review). Results were compared with the standard review procedure, in which mammograms showing interval cancers were mixed with normal mammograms and read in a panel of 17–20 interval cancers per 80 normal studies by radiologists who were aware that they were participating in a review process (ie, informed review).
Results: Of 21 interval cancers, six (29%) were interpreted as positive more often by the informed radiologists than by the uninformed radiologists. For 14 (67%) cancers, there was no difference in detection rate between the two groups, and one cancer (5%) was seen by one of the uninformed radiologists but by none of the informed radiologists. The screening environment review process was found to be feasible at the low volumes tested.
Conclusion: The number of false-negative cancers was higher in the informed review than in the uninformed review. This result suggests that bias exists with the informed review process.
© RSNA, 2007
|
INTRODUCTION
|
|---|
An interval cancer is defined as a cancer diagnosed clinically after screening mammograms have been deemed negative. In some programs, this definition may be refined to specify a time period within which the cancer must be diagnosed for it to be considered an interval cancer. For our purposes, this time period was 24 months.
In some instances, the cancer may not be visible, even retrospectively. These cancers are deemed true-negative. False-negative cancers are those for which findings were thought to be negative but cancers were visible in retrospect. The manner in which these cancers are reviewed varies according to context and can influence whether the cancer is deemed visible in retrospect (1–5). The most extreme example is in the context of legal action, in which a plaintiff's attorney asks a radiologist to act as an expert witness. Bias is unavoidable in this context. The radiologist knows that the mammogram is likely the basis for potential legal action because an attorney made the request. Ideally, the radiologist should attempt to reproduce, as realistically as possible, the circumstances in which the defendant radiologist first viewed the mammogram (6); however, this may not always be possible.
Since 1988, our screening mammography program has called for all interval cancers to be reviewed with what has been described as a "blind review." A pilot project that was performed earlier and involved uninformed reviews proved unsuccessful, as some of the radiologists identified the inserted mammograms; thus, blinding was unsuccessful. Once this became apparent, we developed an informed review system that was considered the next best option.
In our screening program, we have the opportunity to learn of all cancers in the region by means of linkage with the regional breast tumor registry. The standard "next best" informed blind review system of this program is described below. It was understood that this process might inadvertently introduce a bias toward abnormal calls; however, the screeners' advisory committee determined that this was the most practical alternative for reviews. Recently, as more radiologists have had the opportunity to review interval cancers in the context of the interval cancer data for the program, there has been an incentive to reassess the current review system. The radiologists who perform next best informed blind reviews know that they are looking for subtle cancers and that the review set is enriched with cancers; therefore, the review environment does not simulate the screening environment in which the original interpretation was made. Thus, even the current review system may be subject to expectancy bias, which is defined as a tendency for those with knowledge of the proportion of interval cancers present to read with a lower threshold for rating mammographic findings positive. For example, in a typical screening situation, it is expected that there will be between three and seven cancers per thousand screening mammograms. Thus, if a radiologist interprets 100 mammograms in a real screening session, there might not be any cancers or, at most, one or two cancers. The real-life screening situation was contrasted with the informed review mix, in which approximately 17–20 mammograms depicting cancer were mixed with 80 normal mammograms to make a group of about 100 mammograms.
The purpose of our study was to retrospectively assess the feasibility of an uninformed review process to evaluate interval breast cancers and to compare the number of false-negative cancers detected at uninformed review with the number detected at standard informed review.
|
MATERIALS AND METHODS
|
|---|
Screening Mammograms
All the interval cancers from the screening center where the pilot study was performed were selected. All images were film mammograms obtained between September 2002 and November 2004. All mammograms had already been reviewed with the standard process described below. University of British Columbia and British Columbia Cancer Agency institutional review board approval was obtained for this retrospective study; informed consent was waived for use of patient images. When the three radiologists who participated in the uninformed review (ie, they were not aware they were participating in a study) learned of their participation, they consented to the use of their data, provided they would not be identified. Our institutional review board found this acceptable.
Procedures
For the standard informed review, clerical staff members obtained the most recent screening mammogram that was originally read as negative, the findings of any subsequent diagnostic examinations (ie, mammography, ultrasonography) performed since the time of clinical presentation, and the biopsy results. The screening mammograms, including 17–20 images that showed interval cancers, were mounted on a 100-frame multiviewer and mixed with 80 normal mammograms to result in a ratio of approximately four normal mammograms to each mammogram that showed an interval cancer. The mammograms were then independently read by radiologists who volunteered to participate and were instructed to complete report forms as if they were reading studies in a routine screening setting. Each study was supposed to be reviewed by three radiologists; however, the number occasionally varied on the basis of the availability of the radiologists. Records were kept by clerical staff to indicate how radiologists identified the cancer.
The pilot study uninformed review was conducted at a large metropolitan screening center. Three radiologists read mammograms at this center. The clerical staff obtained interval cancer mammograms from the central office after the standard review process was complete. The mammograms were made to look like current studies. The original colored year stickers were removed, and stickers from the current year were applied. The original report forms were temporarily removed from the storage bags. Labels with the current date were printed, and blank report forms were prepared. Studies were mounted and mixed with the daily mammography workload. Typically, 30–60 studies were interpreted each day, and one interval cancer was inserted in each day's workload. Prior studies obtained 2 years earlier, if any, were mounted as per the routine for comparison and were not masked.
When the truly uninformed review was proposed to the academic committee of the screening program, there was initially some skepticism as to whether the studies could be adequately masked so that radiologists would not realize that they were not current studies. Although uninformed reviews had been attempted in the late 1980s and early 1990s, the clerical staff agreed to consider whether it was feasible to try to perform truly uninformed reviews again, realizing that it would be time-consuming for the staff. The matter was set aside, and the radiologists who had proposed the trial were not aware that the clerical staff had taken it upon themselves to set up the process for uninformed reviews. It was only after the study was complete that the radiologists became aware that they had performed truly uninformed reviews. Thus, the studies were read according to each radiologist's preference. Two of the three radiologists read studies in daily batches of 30–60 images. One of the radiologists preferred to read less frequently and read two or more batches per sitting. Clerical staff members kept records of how each of the three radiologists read each study in the truly uninformed blind review. For confidentiality purposes, results for each radiologist are not presented. The results were then compared with the results of the informed review performed by other radiologists (see below). For each radiologist, regardless of whether he or she participated in informed or uninformed review, a positive diagnosis was counted only if it was made at the site where cancer was eventually diagnosed. As in any screening setting, there were some cancers diagnosed when there was a finding on only one view.
Radiologists
The three radiologists who performed the uninformed reviews each had extensive experience in screening mammography and had participated in the regional screening program for 11–18 years. At the time of the study, each radiologist was interpreting 3127–6073 screening mammograms, as well as a variable number of diagnostic mammograms, each year. The other two or three radiologists who performed informed reviews had 6–15 years of experience and were reading 2000–3900 screening mammograms each year.
Statistical Analysis
For each of the two review methods, findings were classified as true-negative if none of the reviewing radiologists identified the cancer and as false-negative if any of the reviewers identified the cancer. Records of whether one, two, or three reviewers identified the cancer were kept. No test of statistical significance was applied to this small data set.
|
RESULTS
|
|---|
Twenty-one mammograms showing interval breast cancers were reviewed. Of these, 11 were deemed negative by all two or three informed radiologists and all three uninformed radiologists. An equal number of radiologists in both the informed and uninformed reviews classified three mammograms as abnormal. For a total of 14 (67%) cancers, there was no difference in detection rate between the two review methods. One cancer (5%) was seen by one of the uninformed radiologists but was not seen by any of the informed radiologists (Table, see case 1). Six cancers (29%) were seen (that is, they were false-negative cancers originally) more often by informed radiologists than by uninformed radiologists (see cases 2–7).
|
DISCUSSION
|
|---|
Of 21 interval cancers reviewed, six (29%) were read as positive more often by the informed radiologists than by the uninformed radiologists compared with one (5%) seen more often by uninformed radiologists. The true screening environment uninformed review process appears to be feasible at the low volume tested (21 cancers).
We believe that the standard informed blinded review process is subject to more bias because the participating radiologists expect to find interval cancers in the enriched set of studies and may therefore diagnose a subtle finding as abnormal more frequently than they would in routine daily interpretation. Our true uninformed review process was more complicated to conduct than the standard process because the studies had to be masked to look like current ones. Studies used for our pilot project were not technically different from those obtained currently.
The time required for uninformed reviews was not problematic; however, uninformed reviews took far longer to perform than standard informed reviews. Radiologists rotate on a weekly basis; thus, it took 21 working days for each radiologist to review all 21 interval cancers in an uninformed manner at the rate of one interval cancer per day. It took 63 working days for all three radiologists to complete the uninformed review. For the standard informed review process, 21 interval cancers can be read in one sitting when mixed with 80 normal mammograms to total 100 studies on the multiviewer. Although we recognize that this method may not be feasible in some screening settings, we did not find substantive problems that would argue against use of this method to review interval cancers in a large high-volume film-based screening environment, because the numbers of interval cancers are relatively small. We cannot comment on the logistics of performing a similar study in a digital environment.
The influence of review design on percentages of missed interval breast cancers has been examined by other authors. Most recently, Hofvind et al (3) reported that the percentage of interval cancers classified as missed ranged from 1.3% to 35.9% depending on the review design. The designs they examined included mixed blinded review (both individual and paired) and informed review (again, both individual and paired). In their informed review, radiologists interpreted screening and diagnostic mammograms, as well as pathology and surgical reports. Their mixed blinded review was similar to our standard review. We noted that the retrospective detection rate was likely affected by the difference in setting between a review and usual screening, as well as the different ratio of cancer-positive findings to cancer-negative findings (1:1.1 in their reviews and 1:167 in their screening). They commented that the mixing of mammograms in a review ideally should be similar to that in the screening setting.
The difficulty of judging retrospectively whether a cancer has been missed is reflected in the literature on negligence litigation. The potential influence of bias is most notable in instances where attorneys ask radiologists for their expert opinion. The importance of environmental context is emphasized by the following statement from a decision rendered by the appellate court of Wisconsin, as noted in Berlin's discussion of bias (6):
In determining whether a physician was negligent, the question is not whether a reasonable physician, or an average physician, should have detected the abnormalities, but whether the physician used the degree of skill and care that a reasonable physician, or an average physician, would use in the same or similar circumstances. [Emphasis added.]
Prior to the 1990s, most reports of false-negative rates for screening mammography involved retrospective reviews of interval cancers. In 1993, Harvey and colleagues (7) studied impalpable breast cancers, comparing standard blinded review (without knowledge that carcinoma was subsequently detected) with retrospective review (with the mammogram showing the carcinoma for comparison). The results showed that twice as many (n = 88) retrospective interpretations were positive, compared with 42 in the blinded review. In 1999, Moberg and colleagues (2) compared two models for reviewing interval cancers and reported that the proportion of missed cancers varied between 7% and 34%, depending on the review method used. Mixed reviewing (interval cancers mixed with other screening images in a ratio of 1:8) reduced the number of cancers identified as missed by 50% compared with nonmixed reviewing. These results are consistent with our findings, showing that the method of review can substantially affect the number of interval cancers identified.
A potential limitation of our study is that the actual dates of the screening examinations were not changed on the film hard copies. These dates, along with the demographic data and the technical information that is automatically recorded on some of the mammograms, are part of the flasher, and these data cannot be changed without either cutting the film hard copy or covering the area of the flasher with an adhesive paper label. During the course of this pilot project, however, none of the radiologists raised concern that they were reading anything other than the daily workload. The studies are marked with colored year stickers and mounted for batch reading on a multiviewer, and the storage bags and matching blank reports are stacked in order. Each bag is numbered to match the panel on the multiviewer, which holds the corresponding patient's study. Thus, the radiologists check to ensure that the number on the report form matches the frame on the multiviewer. Even if the name is checked, it is unlikely that the radiologist will look for the date on a flasher, since there is a colored date sticker on the mammogram. Another potential limitation of the study was the use of two different sets of radiologists for the informed and uninformed reviews. Although equivalency is difficult to prove, all radiologists in our screening program must pass a standardized screener test and must meet continuing medical education requirements.
Our conclusions are limited by the small sample size of 21 cancers in this preliminary pilot study. We are encouraged, however, by the ease with which the studies were inserted into the daily screening workload. This process may not be applicable to all screening settings and may not be possible when screening mammograms are in digital format; however, where it is feasible, we believe it will improve the validity of review results and provide a context more similar to the one in which the original radiologist had to work.
These study results quantify what seems intuitively obvious to radiologists who read screening mammograms. Whenever the interpreting radiologist is aware of a higher-than-usual possibility of cancer, it can influence the interpretation such that the study is more likely to be interpreted as positive, consistent with the lower threshold for interpretation that was discussed in other articles. Of course, this concept could apply to the involvement of expert witnesses in the current tort system. It would be ideal if expert testimony was based on blinded reviews by two or more uninformed radiologists to determine in a more unbiased way whether cancers could have been detected at the initial prospective interpretation.
|
ADVANCES IN KNOWLEDGE
|
|---|
- The number of false-negative cancers is higher when radiologists know they are reviewing studies that contain interval cancers than when they do not know they are reviewing studies that contain interval cancers.
- Reviewing interval cancers in a true uninformed screening environment is feasible at low interval cancer volumes.
|
ACKNOWLEDGMENTS
|
|---|
The authors acknowledge Dayna MacDonald of the Vancouver Screening Centre at British Columbia Women's Hospital for her work in masking the film hard copies and inserting them into the daily workload. We thank Elaine Simpson of the Screening Mammography Program of British Columbia for her kind help with the retrieval of film hard copies for the interval cancer cases.
|
FOOTNOTES
|
|---|
Guarantors of integrity of entire study, P.B.G., M.J.B.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, all authors; clinical studies, P.B.G., L.J.W.B.; statistical analysis, M.J.B.; and manuscript editing, all authors
Authors stated no financial relationship to disclose.
|
References
|
|---|
- Moberg K, Grundstrom H, Lundquist H, Svane G, Havervall E, Muren C. Radiological review of incidence breast cancers. J Med Screen 2000;7:177–183. [Abstract/Free Full Text]
- Moberg K, Grundstrom H, Tornberg S, et al. Two models for radiological reviewing of interval cancers. J Med Screen 1999;6:35–39. [Abstract/Free Full Text]
- Hofvind S, Skaane P, Vitak B, et al. Influence of review design on percentages of missed interval breast cancers: retrospective study of interval cancers in a population-based screening program. Radiology 2005;237:437–443. [Abstract/Free Full Text]
- de Rijke JM, Schouten LJ, Schreutelkamp JL, Jochem I, Verbeek AL. A blind review and an informed review of interval breast cancer cases in the Limburg screening programme, the Netherlands. J Med Screen 2000;7:19–23. [Abstract/Free Full Text]
- Duncan AA, Wallis MG. Classifying interval cancers. Clin Radiol 1995;50:774–777. [CrossRef][Medline]
- Berlin L. Hindsight bias. AJR Am J Roentgenol 2000;175:597–601. [Free Full Text]
- Harvey JA, Fajardo LL, Innis CA. Previous mammograms in patients with impalpable breast carcinoma: retrospective vs blinded interpretation. AJR Am J Roentgenol 1993;161:1167–1172.
TOP
ABSTRACT
INTRODUCTION
