HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) All Versions of this Article: 2241011470v1 224/1/42    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Yoon, L. S. Articles by Forman, H. P. Search for Related Content PubMed PubMed Citation Articles by Yoon, L. S. Articles by Forman, H. P.

Evaluation of an Emergency Radiology Quality Assurance Program at a Level I Trauma Center: Abdominal and Pelvic CT Studies¹

¹ From the Departments of Diagnostic Radiology (L.S.Y., A.H.H., J.A.B., H.P.F.), Trauma Surgery (R.R.), and Economics (H.P.F.), and the School of Management (H.P.F.), Yale University School of Medicine, 333 Cedar St, 2-332 SP, New Haven, CT 06520. From the 2001 RSNA scientific assembly. Received September 5, 2001; revision requested October 10; revision received November 27; accepted January 7, 2002. Address correspondence to H.P.F. (e-mail: howard.forman@yale.edu).

	ABSTRACT

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
PURPOSE: To evaluate the use of a redundant system in improving quality of care in the trauma setting by examining a subset of our quality assurance program.

MATERIALS AND METHODS: Five hundred thirty-one consecutive abdominal and pelvic CT reports obtained in patients with trauma at a level I trauma center from August 22, 1999, to August 21, 2000, were retrospectively reviewed. Each case was initially interpreted by a board-certified or board-eligible radiologist during evaluation in the emergency department and was subsequently reviewed by a subspecialty abdominal imaging radiologist as part of a quality assurance program. Nineteen cases were excluded because available information was incomplete, resulting in 512 cases in the current study. Cases with discordant interpretations were followed up to discern care change.

RESULTS: Of the 512 trauma cases, 153 (29.9%) showed discordant readings. Review of patient records demonstrated changes in patient care in 12 (7.8%) cases. Three (2.0%) cases were reviewed from the morbidity and mortality records of the Department of Trauma Surgery as a direct result of misinterpretations. Six (4%) cases involved additional diagnostic imaging for reevaluation; in four of these six cases the quality assurance reader’s interpretation was confirmed, while in the other two, the initial interpretations were favored.

CONCLUSION: Findings suggest that discordant radiologic interpretations most often do not result in a change in patient care and outcome. The quality assurance program did, however, identify and lead to changes in care in a number of cases by providing clinically important additional findings.

© RSNA, 2002

Index terms: Abdomen, CT, 70.1211 • Emergency radiology • Pelvis, CT, 80.1211 • Quality assurance

	INTRODUCTION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
In 1999, the Institute of Medicine released a report in which it was estimated that up to 98,000 Americans die each year as a result of preventable medical errors (1). Errors were defined as "the failure to complete a planned action as intended or the use of a wrong plan to achieve an aim; not all errors result in harm" (1). Since 1999, there have been numerous reports (2–5) in which authors have exposed, addressed, and recommended ways to reduce medical errors. One of the key areas relates to improvement of reporting systems and use of technologic advances.

Medical errors occur in various settings; among the errors that may affect patients undergoing imaging are those due to misinterpretation. Radiologic errors due to missed diagnoses, however, are often difficult to ascertain, since observer variation in interpretation does not necessarily represent medical error (6). Investigators in several studies (7–13) have examined the frequency and clinical consequences of misinterpretations in the emergency department setting. These studies have shed much light on the effectiveness of various quality assurance (QA) programs. All of these studies, however, included comparisons of interpretations by radiology residents, fellows, or nonradiologist attending physicians with those by attending radiologists but did not include comparisons of interpretations between attending radiologists.

One of the characteristics of highly reliable industries includes high levels of redundancy in personnel and safety measures (1). To achieve this, in April 1999 our institution established a new QA system that complemented our existing 24 hours per day, 7 days per week coverage by an attending radiologist in the emergency department. Currently, findings of every nonconventional radiologic patient examination in the emergency department are interpreted by the attending radiologist in the emergency department and are subsequently reviewed by a subspecialty attending radiologist in the next 24 hours. During the first 2 years of this QA program, no formal analysis of the value of this approach had been undertaken.

We hypothesized that clinically important improvement of patient care and outcome has occurred with use of our QA program. The purpose of our study, therefore, was to evaluate the use of a redundant system in improving the quality of care in the trauma setting by examining a subset of our QA program.

	MATERIALS AND METHODS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
We retrospectively reviewed 531 consecutive abdominal and pelvic computed tomographic (CT) reports in patients with trauma who underwent evaluation in an urban university-affiliated level I trauma center from August 22, 1999, to August 21, 2000. Nineteen cases were excluded due to available information being incomplete, resulting in 512 cases in our study. As mentioned, since April 1999, in accordance with our QA program, every nonconventional radiologic report (for CT, magnetic resonance [MR] imaging, ultrasonography [US]), and nuclear medicine) in a patient in the emergency department has received a preliminary interpretation. This interpretation is performed by an attending radiologist, the "primary reader," in the emergency department, and a secondary review is performed by a subspecialty attending radiologist, the "QA reader," within 24 hours after the initial interpretation. The original report is generated with use of a voice recognition system during primary reader evaluation, thus allowing immediate generation of a hard-copy report. The QA report is generated by using handwritten comments on a copy of the original report and includes the QA reader’s initials. The report is then returned to the primary reader for a second review. At the discretion of the primary reader, the report receives an addendum. When there is major discordance, the QA reader immediately contacts the primary reader if he or she is available. The case is discussed, and the clinicians are subsequently contacted. If the primary reader cannot be reached, the QA reader contacts the clinicians immediately and may amend the report. All QA reports are archived when the rechecking process is complete. The radiology information system (IDX Rad; IDX, Burlington, Vt) archives final reports but does not incorporate information about the rechecking physician or his or her comments into the electronic record. The 512 consecutive abdominal and pelvic CT examinations in the current study represented a subset of our overall data and included both adult and pediatric patients seen in the emergency department. This study was approved by the human investigation committee of our institution. Informed consent was not required by the committee for this study.

For each case, patient name, age, and sex; clinical indications; names of primary and QA readers; traumatic abdominal and pelvic findings; traumatic extraabdominal and pelvic findings; and incidental findings were obtained and recorded. All 512 reports were then divided into two main categories: (a) those with complete concordance of interpretations and (b) those with discordance of interpretations. Findings identified by the QA reader and handwritten on the original report were considered discordant. Comments by the QA reader regarding anatomic variations (eg, normally sized retroperitoneal nodes or retroaortic left renal veins) or incidental observations (eg, tampon in the vagina; correctly placed nasogastric tube or Foley catheter) were not considered to indicate a discordant reading. The primary readers (including H.P.F. and A.H.H.) consisted of 21 board-certified or board-eligible radiologists who have varying years of experience ranging from less than 1 year to more than 20 years. Five of the 21 primary readers were trained through a body-CT fellowship. One primary reader had specific training in emergency radiology. The QA readers were 18 subspecialty radiologists who also have varying years of experience ranging from less than 1 year to more than 20 years. The QA readers reviewed all cases, regardless of the training background of the primary reader.

For the reports with discordant interpretations, additional data were obtained by means of (a) review of the patient medical record, (b) review of the correlated record of the department of trauma surgery morbidity and mortality conferences, and (c) reevaluation of the final imaging reports and additional imaging reports. First, patient medical records for all cases with discordant interpretations were obtained and reviewed to determine the clinical importance of these interpretations. Readmission and new surgery, treatment, or diagnostic examinations (both imaging and laboratory) resulting from discordant second readings were considered clinically important. Second, records of the Department of Trauma Surgery morbidity and mortality conferences were used to match morbidity and mortality cases with discordant interpretations. Cases with a positive match were further reviewed to determine whether morbidity and/or mortality resulted from radiologic misinterpretation or other unrelated issues. Third, the final diagnostic imaging reports for all cases with discordant interpretations were obtained, reviewed, and classified as "no change," "edited," or "with an addendum." For each case, subsequent imaging reports were reviewed by using the radiology information system, and new imaging studies performed because of discordant readings were used to determine whether the preliminary or QA interpretation was accurate.

	RESULTS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
Of the 512 trauma cases, 153 (29.9%) showed discordant readings between the preliminary interpretation by the attending radiologists in the emergency department (primary readers) and the QA review by the subspecialty abdominal imaging radiologists (QA readers). The 512 reports from the former cases comprised only abdominal and pelvic CT examinations performed at the initial emergency department encounter for which information on QA reader identification and comments was complete.

Review of all 153 cases demonstrated that patient care was changed in 12 (7.8%). One readmission occurred after a patient received a diagnosis of adrenal hemorrhage from the QA reader. This patient, who was contacted at home, was discharged again after subsequent physical examination and laboratory tests showed no sequelae. Other changes in patient care included laboratory evaluation for suspected cirrhosis, orthopedic and pain management consultations for presumed acute vertebral compression fracture, esophagography to rule out esophageal perforation for pneumomediastinum, gastroenterologic consultation with regard to thickened small bowel, and serial hematocrit level checks for suspected hepatic laceration (Table 1).

Review of the final diagnostic imaging reports in the discordant cases demonstrated (a) no change to preliminary reports in 95 (62.1%) cases, (b) changes incorporated into final reports in 27 (17.6%) cases, and (c) addenda to final reports in 31 (20.3%) cases. Furthermore, review of subsequent diagnostic imaging reports for the discordant cases showed that six (3.9%) had undergone additional diagnostic imaging for reevaluation. Although the QA reader recommended various follow-up imaging examinations for reevaluation in 13 cases, follow-up examinations were performed in only six. These, however, included only procedures performed at our institution. As shown in Table 2, in four of the six cases the findings of the QA reader were confirmed: two hepatic lacerations, one instance of bowel loops instead of anomalous veins, and one rib fracture with a hemorrhagic renal cyst. In the remaining two cases, the initial interpretation was favored. In both cases, bladders with suspected injury according to the QA reader actually turned out to be normal according to the follow-up CT cystogram.

	DISCUSSION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

Authors of previous studies (7–13) have investigated the subjects of radiologic errors in general and the frequency and clinical consequences of radiologic misinterpretations in a trauma setting. The average error rate among radiologists has been around 30%, according to studies dating from 1949 to 1992 (14–16). Although many errors are due to disagreement in interpretations and often do not result in a change in clinical management and outcome, some of the missed findings result in unfavorable clinical consequences. Such missed findings often have far-reaching ethical and legal consequences, and the ethical and medicolegal considerations of radiologic errors have been the subjects of ongoing debate for many years (14,17). Moreover, investigators in several studies have examined the occurrence and clinical consequences of radiologic errors, specifically in the emergency department. Wechsler et al (7) compared the preliminary interpretation of emergency body CT scans by residents or fellows with the secondary review by attending radiologists and showed that major discordance occurred in 1.2% (seven of 597 studies) and minor discordance occurred in 6.5% (39 of 597 studies). Eachempati et al (8) evaluated alterations of preliminary readings in the emergency department and their effect on patients with trauma. In this study, as well, preliminary interpretations by radiology residents were compared with secondary reviews by attending radiologists; it was concluded that alterations of preliminary readings minimally affect outcomes in patients with trauma. Authors of other studies (9–13) have also investigated the frequency and clinical consequences of radiologic errors in the emergency department. All of these studies (7–13) have, however, involved comparison of interpretations by radiology residents, fellows, or nonradiologic attending physicians with those by attending radiologists; no comparisons were made between attending radiologists. The main objective of these studies was to determine the effect of training and experience on radiologic interpretations.

The current study is different from previous investigations in that we compared interpretations between attending radiologists, focusing on one subset of our QA program: abdominal and pelvic CT examinations in patients with trauma. Our discordance rate of 29.9% (153 of 512 cases) is higher, as compared with that in the previous study by Wechsler et al (7). There are, however, important differences between these two studies. In our study, retrospective review of discordant interpretations was performed between attending radiologists, whereas Wechsler et al (7) prospectively examined discordant interpretations between residents or fellows and attending radiologists. Although 153 of the 512 cases in the current study had discordant interpretations, only 12 of those 153 cases resulted in perceived changes in patient care. One case was of major concern, as the patient needed to return to the emergency department for reexamination. In the other 11 cases, additional diagnostic studies, laboratory values, new medications for pain and possible infection, and specialist consultations were required. It is also important to note that three of the 12 cases had clinically important nontraumatic findings: suspected cirrhosis, an inflammatory small-bowel process, and suspected ovarian cystadenoma. In the remaining 141 cases, new findings by the QA reader did not affect the clinical care of the patients.

The other important finding involved follow-up of the discordant cases. Of these 153 cases, only 58 (37.9%) showed changes (edits, addenda) in the final reports. The remaining 95 (62.1%) had identical preliminary and final reports. This finding suggests that the primary reader, more often than not, found the QA reader’s suggestion to be not clinically important enough to warrant changes to the original report. Furthermore, in six cases, review of subsequent diagnostic imaging reports for reevaluation allowed us to ascertain whether the consensus lay with the primary reader or with the QA reader. These additional imaging examinations were performed at the QA reader’s recommendation if the original reports raised any suspicion of abnormal findings that were not adequately identified initially. In four of the six cases, findings of the subsequent examinations agreed with the QA reader’s interpretation. In the remaining two cases, the initial interpretation was favored.

Our findings bring to light two important issues. First, our QA program serves an important purpose in identifying clinically important, however infrequent, findings missed by the primary reader. The demonstration of changes in patient care suggests that the line of communication between the QA reader, primary reader, and clinician serves to improve patient care when needed. Second, despite the high rate of discordant interpretations (29.9%) in the current study, most are not clinically important and do not result in a change in patient care. Only 7.8% (12 of 153) of the discordant readings and 2.3% (12 of 512) of the total readings resulted in a change in patient care.

There were several limitations of this study. First, compliance with our QA system among the primary and QA readers was not perfect. Nineteen cases were excluded from our data set because there was no QA reader name or comment. At our review of these records, there was no indication that these cases represented errors, since, at the time this article was written, no follow-up imaging had occurred and no mention had been made in the records of the Department of Trauma Surgery. Still, we cannot confirm what the QA findings would have been at that time. Second, the rate of accuracy of interpretations, as measured against an infallible standard of reference, were not provided. Although the QA reader, who has specialty training in body imaging, was often more experienced in reading body CT scans than the primary reader and was certainly working in a better setting for interpretation, in at least two cases the final interpretation favored the primary reader. Third, since this study focused on patients with trauma in the emergency department setting, many of the recommendations made by the QA reader for further examinations often were not followed up. After the patient is discharged, it is often difficult to contact and bring the patient back for further examinations (8). Fourth, use of medical records to identify cases in which a change in clinical management occurred may have been potentially biased by reviewer subjectivity.

To streamline our QA process, the Department of Diagnostic Radiology has recently hired a QA coordinator to oversee our QA program, as well as to ensure that the process includes all cases, with appropriate documentation of the QA reader’s findings and name. Other efforts include encouraging stricter compliance by the attending radiologists and reducing lag time between initial interpretation and QA review. Furthermore, it is expected that the findings of such a program will eventually include proposals for remediation or continuing medical education if a given primary reader is found to be deficient in an area of required expertise.

Another concern regarding the QA program pertains to liability. Although 153 of the 512 cases involved discordant interpretations, it would be presumptuous to label them as 153 errors. Many of these 153 cases were the result of incidental additional findings of minimal clinical consequence. As Berlin (14) has suggested, radiologists need to be careful when describing their findings, and words such as missed, error, or mistake should be avoided in official reports.

Our findings show that clinically important improvement of patient care occurs with a QA program in which redundant systems are used. Although most discordant interpretations do not result in change in patient care, there are a number of cases in which patients are managed differently as a result of new clinically important findings. As identification and reduction of medical errors become increasingly important in health care, evaluation of existing QA programs such as ours will serve a useful purpose in monitoring the effectiveness of the current system and in making necessary changes to improve the system. Moreover, we believe that it will provide an invaluable educational experience for the house staff members and attending radiologists as they learn from discordant interpretations, as well as from actual errors. Future studies to classify each discordant finding into various categories may provide additional insight into our QA program and the study of medical errors in general.

	FOOTNOTES

 
Abbreviation: QA = quality assurance

Author contributions: Guarantor of integrity of entire study, H.P.F.; study concepts, H.P.F., A.H.H., J.A.B.; study design, all authors; literature research, L.S.Y.; clinical studies, L.S.Y., A.H.H.; data acquisition, L.S.Y.; data analysis/interpretation, L.S.Y., H.P.F.; statistical analysis, L.S.Y.; manuscript preparation, L.S.Y., H.P.F.; manuscript definition of intellectual content, H.P.F., A.H.H.; manuscript editing, J.A.B., H.P.F., A.H.H., R.R.; manuscript revision/review, H.P.F.; manuscript final version approval, all authors.

	REFERENCES

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 

1. Institute of Medicine. To err is human: building a safer health system Washington, DC: National Academy Press, 1999.
2. Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century Washington, DC: National Academy Press, 2001.
3. Quality Interagency Coordination. Doing what counts for patient safety: federal actions to reduce medical errors and their impact. Washington, DC: Report of the Quality Interagency Coordination Task Force to the President, February 2000.
4. Agency for Healthcare Research and Quality. Translating research into practice: reducing errors in health care. Available at: www.ahrq.gov/research/errors.htm 2001; Accessed September 1.
5. Agency for Healthcare Research and Quality. Medical errors: the scope of the problem. Available at: www.ahrq.gov/qual/errback.htm 2001; Accessed September 1.
6. Robinson PJ. Radiology’s Achilles’ heel: error and variation in the interpretation of the Roentgen image. Br J Radiol 1997; 70:1085-1098.[Abstract]
7. Wechsler RJ, Spettell CM, Kurtz AB, et al. Effects of training and experience in interpretation of emergency body CT scans. Radiology 1996; 199:717-720.[Abstract/Free Full Text]
8. Eachempati SR, Flomenbaum N, Seifert C, Fischer E, Hydo LJ, Barie PS. Alterations of preliminary readings on radiographic examinations minimally affect outcomes of trauma patients discharged from the emergency department. J Trauma 2000; 48:654-658.[Medline]
9. Lal NR, Murray UM, Eldevik OP, Desmond JS. Clinical consequences of misinterpretations of neuroradiologic CT scans by on-call radiology residents. AJNR Am J Neuroradiol 2000; 21:124-129.[Abstract/Free Full Text]
10. Walsh-Kelly CM, Melzer-Lange MD, Hennes HM, et al. Clinical impact of radiograph misinterpretation in a pediatric ED and the effect of physician training level. Am J Emerg Med 1995; 13:262-264.[CrossRef][Medline]
11. Klein EJ, Koenig M, Diekema DS, Winters W. Discordant radiograph interpretation between emergency physicians and radiologists in a pediatric emergency department. Pediatr Emerg Care 1999; 15:245-248.[Medline]
12. Lufkin KC, Smith SW, Matticks CA, Brunette DD. Radiologists’ review of radiographs interpreted confidently by emergency physicians infrequently leads to changes in patient management. Ann Emerg Med 1998; 31:202-207.[CrossRef][Medline]
13. Roszler MH, McCarroll KA, Rashid T, Donovan KI. Resident interpretation of emergency computed tomographic scans. Invest Radiol 1991; 26:374-376.[CrossRef][Medline]
14. Berlin L. Reporting the "missed" radiologic diagnosis: medicolegal and ethical considerations. Radiology 1994; 192:183-187.[Free Full Text]
15. Garland LH. On the scientific evaluation of diagnostic procedures: presidential address—34th annual meeting of the Radiological Society of North America. Radiology 1949; 52:309-328.[Medline]
16. Renfrew DL, Franken EA, Berbaum KS, Weigelt FH, Abu-Yousef MM. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992; 183:145-150.[Abstract/Free Full Text]
17. Berlin L. Does the "missed" radiographic diagnosis constitute malpractice? Radiology 1977; 123:523-527.[Abstract]

This article has been cited by other articles:

				C. G. Filippi, B. Schneider, H. N. Burbank, G. F. Alsofrom, G. Linnell, and B. Ratkovits Discrepancy Rates of Radiology Resident Interpretations of On-Call Neuroradiology MR Imaging Studies Radiology, December 1, 2008; 249(3): 972 - 979. [Abstract] [Full Text] [PDF]

				R. B. Ruchman, J. Jaeger, E. F. Wiggins III, S. Seinfeld, V. Thakral, S. Bolla, and S. Wallach Preliminary Radiology Resident Interpretations Versus Final Attending Radiologist Interpretations and the Impact on Patient Care in a Community Hospital Am. J. Roentgenol., September 1, 2007; 189(3): 523 - 526. [Abstract] [Full Text] [PDF]

				O. Israel and A. Kuten Early Detection of Cancer Recurrence: 18F-FDG PET/CT Can Make a Difference in Diagnosis and Patient Care J. Nucl. Med., January 1, 2007; 48(1_suppl): 28S - 35S. [Abstract] [Full Text] [PDF]

				S. P. Sheedy, F. Earnest IV, J. G. Fletcher, J. L. Fidler, and T. L. Hoskin CT of Small-Bowel Ischemia Associated with Obstruction in Emergency Department Patients: Diagnostic Performance Evaluation Radiology, December 1, 2006; 241(3): 729 - 736. [Abstract] [Full Text] [PDF]

				D. D. Saketkhoo, M. Bhargavan, J. H. Sunshine, and H. P. Forman Emergency Department Image Interpretation Services at Private Community Hospitals Radiology, April 1, 2004; 231(1): 190 - 197. [Abstract] [Full Text] [PDF]

				E. Carney, J. Kempf, V. DeCarvalho, A. Yudd, and J. Nosher Preliminary Interpretations of After-Hours CT and Sonography by Radiology Residents Versus Final Interpretations by Body Imaging Radiologists at a Level 1 Trauma Center Am. J. Roentgenol., August 1, 2003; 181(2): 367 - 373. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) All Versions of this Article: 2241011470v1 224/1/42    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Yoon, L. S. Articles by Forman, H. P. Search for Related Content PubMed PubMed Citation Articles by Yoon, L. S. Articles by Forman, H. P.