Radiology
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

DOI: 10.1148/radiol.2412051400
This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Kruskal, J. B.
Right arrow Articles by Kressel, H. Y.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Kruskal, J. B.
Right arrow Articles by Kressel, H. Y.
(Radiology 2006;241:518-527.)
© RSNA, 2006

Special Reports

Implementation of Online Radiology Quality Assurance Reporting System for Performance Improvement: Initial Evaluation1

Jonathan B. Kruskal, MD, PhD, Chun S. Yam, PhD, Jacob Sosna, MD, Donna T. Hallett, BS, RT, Yolanda J. Milliman, RN and Herbert Y. Kressel, MD

1 From the Department of Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215. From the 2005 RSNA Annual Meeting. Received August 29, 2005; revision requested October 13; revision received December 5; accepted January 6, 2006; final version accepted January 9. Address correspondence to J.B.K. (e-mail: jkruskal{at}bidmc.harvard.edu).


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
Purpose: To evaluate an online system developed and implemented for reporting and managing quality assurance (QA) events occurring in a radiology department.

Materials and Methods: This HIPAA-compliant study had institutional review board approval; informed consent was not required. Using repeated plan-do-study-act cycles, a radiology quality management team applied a 10-step process to implement an online reporting system. The system permits remote submission of cases by staff members. The number of weekly submissions to the system over a 9-month period was evaluated and compared with that for the preceding 6 months by using the Mann-Whitney test. Sources and nature of data, actions initiated, and final outcomes were also analyzed. Recorded data included forum of discussion, dimension of care, action items, monitoring of follow-up and compliance, and notification status.

Results: During the first 9 months of implementation, 605 cases were submitted (mean, 21.4 cases per week), a significant increase (P < .005) compared with the preceding 6 months (mean, 3.2 cases per week). Cases, which were submitted by residents (121 cases [20.0%]), fellows (94 cases [15.5%]), faculty members (319 cases [52.7%]), or technologists (54 cases [8.9%]), reported technical (33.1%) or administrative (8.0%) issues. The 329 clinical cases (54.4%) included 60 errors in communication (18.2%), 67 errors in interpretation (20.4%), 78 diagnostic delays (23.7%), 99 missed diagnoses (30.1%), and 54 procedural complications (16.4%); some cases were in more than one category. Twenty-three cases (3.8%) resulted in submission-related QA projects, and 69 cases (11.4%) resulted in individuals or sections of the hospital undergoing additional training.

Conclusion: A secure online QA reporting system promotes reporting of QA events and serves as a database for identifying and managing trends, initiating performance improvement projects, and providing feedback to staff members who submit cases.

© RSNA, 2006


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
Deficiencies in the quality and safety of medical care delivered in this country have been reported (1–8). In response, major efforts are being made to define performance metrics in many clinical fields, including radiology (9). Few data currently exist that define quality and safety metrics related to the practice of radiology and to radiologists. Indeed, objective diagnostic benchmarks for acceptable levels of missed diagnoses are needed (10).

For identification of barriers to improving the quality of radiology services, system rather than individual factors are likely to be the root cause of most problems (11–14). To our knowledge, root cause analyses of radiologic quality initiatives have not been widely employed or described. As Thrall (1) has highlighted, these barriers include poor or nonexistent medical information and management systems—barriers that are further compounded by the underreporting of adverse events.

The Joint Commission on Accreditation of Healthcare Organizations requires that all hospitals develop systems for the ongoing monitoring of clinical and technical performance. This is especially challenging for the field of radiology, where limited outcome data exist. To develop such systems, it is first necessary to establish a simple means for confidential reporting of quality-related matters. This reporting system should be searchable, provide ongoing data for trend identification and analysis, and encourage all staff members—be they technical, administrative, or clinical—to submit events in a manner that results in constructive feedback, is of educational benefit, and encourages identification of improvement priorities.

Physicians may be reluctant to report adverse clinical events, especially if this event represents self reporting or self incrimination or may have punitive consequences. Deitch et al (15) have shown that radiologists remain skeptical about the usefulness of continuous quality improvement. To encourage submission of all data necessary to improve the quality of care, not only must the system be simple, peer reviewed, nonpunitive, and fair, but there also must be some recognizable positive feedback and benefit to staff members for submitting such data. We hypothesized that a simple online reporting system, with which staff members—including radiologists, technologists, managers, and administrators—were regularly made aware of trending data and of quality improvement initiatives resulting from their submissions, would further encourage ongoing submission of data. Thus, the purpose of our study was to evaluate an online system we have developed and implemented for reporting and managing quality assurance (QA) events occurring in a radiology department.


   MATERIALS AND METHODS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
Our hospital's departments of Health Care Quality and Risk Management approved the development and implementation of the online reporting system in this study, which measured the processes involved in the collection of QA data. Our institutional review board granted us an exemption for this Health Insurance Portability and Accountability Act–compliant study and did not require informed consent.

Using repeated plan-do-study-act cycles, we subscribed to a 10-step process for implementation of an online QA reporting system for performance improvement (Table 1). A radiology quality management team (RQMT) was established and met weekly from April 2004 to June 2004 to analyze and identify ways of improving reporting of QA data. We also aimed to improve clinical and technical performance in our academic hospital–based Department of Radiology, in which approximately 400 000 studies are performed annually. Members of the team included the physician director of radiology QA (J.B.K.), the radiology nurse manager (Y.J.M.), and the radiology director of operations (D.T.H.). Support staff members included a software engineer (C.S.Y.), who performed rapid online integration of all changes as the reporting process was developed, and two radiologists (J.S. and H.Y.K.) with 3 and 4 years of involvement with QA programs, respectively.


View this table:
[in this window]
[in a new window]

  		Table 1. Ten-Step Process Applied to Implementation of Online QA Reporting System for Performance Improvement

 
In developing the online system, the team established five criteria to be met prior to department-wide launching of the online system: (a) ease of data entry through widely available in-house computer terminals and the Internet; (b) confidential data submission and peer review; (c) secure e-mail interface to send submitted data through a Web site to appropriate personnel for review and action; (d) database of all submitted data, including current state of review of each entry; and (e) the ability to search the database for analysis of data and to identify trends.

Preimplementation Planning
Prior to the development and launching of the online system on October 1, 2004, we analyzed the existing preimplementation process of QA reporting, with reference to any procedural steps likely to prevent or limit submission of QA data. We also analyzed the number, origin, and type of submissions over the preceding 6-month period (from April 1, 2004, to September 30, 2004). Data from this analysis were then used to facilitate the design of the online system.

Online Radiology QA Reporting System
The system comprises (a) a screen for data input and submission, (b) a Web-based mechanism for routing each submission for peer review, (c) a screen for documenting findings or actions resulting from this review, and (d) a screen for recording that the final review has taken place by the RQMT, which submits the completed case for data analysis and trending. The workflow of the system is illustrated in Figures 1 and 2.


Figure 1
View larger version (13K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 1: Flowchart shows basic workflow (initial steps that occur after submission of a case) of online QA reporting system. Once a case is submitted, a member of RQMT assigns it to an appropriate reviewer. Once reviewed, the case is returned and reviewed by RQMT. If the team is satisfied with the review and outcome, the submitter is notified in an e-mail that the case has been reviewed and he or she can view the outcome or action items. At times, RQMT may request additional review, data, or actions from the reviewer. Once a case is closed, several steps occur (see Fig 2).

 

Figure 2
View larger version (11K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 2: Flowchart shows steps that occur after review by RQMT of a case submitted to online QA reporting system. Once its members are satisfied with review of a case, RQMT will submit data into a database that is monitored and analyzed to identify trends. Findings from case review may also result in the institution of a performance improvement (PI) action or the initiation of a quality improvement (QI) project. Data resulting from this quality improvement project are then analyzed, and a performance improvement action is introduced.

 
Data Input
The reporting system is Web based and is accessible with a secure log-in, which records the name of the person submitting the data and the date of submission. Log-in privileges are provided to QA directors of other hospital departments to facilitate interdepartmental exchange of quality data that require investigation. To route each submission for review and action, the site then requests the input of hospital section (including ambulatory care sites) and imaging modality (for purposes of this system, a picture archiving and communication system [PACS] is considered an imaging modality). In this way, a problem with computed tomographic (CT) images in PACS would be assigned to the appropriate PACS rather than to the CT manager for review and action. Similarly, a missed pulmonary embolus on a CT scan read by a member of the abdominal imaging group would be reviewed by the QA group for the appropriate section, and an error in interpretation of a CT study performed in the emergency department (ED) would be assigned to the QA directors from both radiology and the ED.

The system is fully compliant with Health Insurance Portability and Accountability Act regulations, and only staff members who had taken and passed an online Health Insurance Portability and Accountability Act compliance course and examination were allowed access.

Additional data recorded include patient medical record number or case-specific accession number; these numbers allow us to directly import finalized reports from our radiology information system. It is also possible to submit non-patient topics; indeed, this section of the site encourages staff members to submit such topics for investigation or response. The site thus serves as a comprehensive and interactive performance improvement repository because each of the submitted topics receives a response generated by the RQMT.

The input sheet then asks for a summary of pertinent issues and asks whether the submission relates to clinical or technical matters or both and whether it is in response to a section or modality review session or follows a review by another department or hospital. Using a series of check boxes, the submitter then classifies a submission according to whether it is equipment, technologist, or staff related. For procedures, the submitter also classifies whether there was a major or minor complication, whether errors in communication or interpretation occurred, and whether diagnosis was delayed or states that the submission is simply for purposes of data collection or trending. Causes of error are classified according to a modification of the system proposed by Renfrew et al (16) into errors of complacency, faulty reasoning, lack of knowledge, "underreading," and poor communication. More than one category could be chosen. In addition, we also documented how each case was brought to the submitter's attention, with options that included self report, report from nonradiologist physician, report from another member of our department, report from a patient (or patient family member); and report from another department, another hospital, or our own Healthcare Quality or Risk Management Divisions.

Data Review
Through frequent Web-based monitoring of the site (at least daily, including weekends), all members of the RQMT were able to assign cases submitted for review to modality managers, section chiefs, section QA officers, or other individuals as appropriate. With this system, cases are automatically assigned by e-mailing a unique and secure link to each case through a series of drop-down menus. When this link is accessed, a new online data sheet appears that includes all of the originally submitted data and provides options for documenting findings and outcomes from the case review. Specific boxes are also available to record such data as whether the case was discussed at section or other QA meetings (and when such meetings were held). For analysis purposes, a dimension-of-care category (safety, efficiency, effectiveness, patient centered, or timeliness) is included, and action items (including disciplinary action items) are also requested. Should follow-up be deemed necessary, the name of the person assigned to follow up the case can be inserted, as well as a date for anticipated resolution. This allows the RQMT to be notified when follow-up has not occurred, and an e-mail is automatically generated to remind the assigned person that a response is outstanding. In addition, when a submission is assigned, the date is recorded and that individual is automatically reminded with an e-mail message if the review has not been returned within 14 days.

Additional information that can be recorded after review includes notification status and final outcome. Notification status options include no notification required or notification of one or more of the following individuals or groups: our chairman, section chiefs or managers, modality chiefs or managers, program directors, the Graduate Medical Education Office, hospital compliance officers, patient care representatives, the Department of Risk Management, and the Department of Health Care Quality. Final outcome is categorized according to whether these data should be kept for trending analysis alone, another department or physician should be notified, or this case should be submitted to other hospital committees for discussion. Once these data are inserted, the reviewer submits the case by clicking a button and it is reflected as reviewed (with a submission date) on the online Web site (Fig 3). The case is then ready for review by the RQMT.


Figure 3
View larger version (16K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 3: Online status screen. In this example, status of each case is color coded to distinguish between new submissions (red), cases that have been assigned and date of assignment (green), cases that have been reassigned after 2 weeks (bold green), cases that have been reviewed (blue), red-flagged cases (bold red), and cases that have been reviewed and closed by the RQMT (black). These cases are simply illustrative examples. The accession number relates to the specific case being reported, allowing us to automatically import the report from that specific case. wet = Preliminary.

 
Red Flag System
An experimental red flag system was incorporated to alert us when certain submitted cases may require expedited attention. These cases appear in bold red letters when the site is accessed for review (Fig 3). At this stage of implementation, and because quality standards are not readily available or published on a regional or national basis, these cases are defined by the RQMT and are automatically uploaded onto the online system from related clinical computing databases. For purposes of this analysis, we currently include any ED case that is unapproved by the attending radiologist for more than 24 hours, any change in an ED preliminary report, any incident report involving the Department of Radiology that is filed online with the Department of Health Care Quality (the latter three examples are automatically uploaded onto the site through our hospital Clinical Computing Center), and any completed review that recommends notification beyond the level of section chief or modality manager, including the Massachusetts Department of Public Health or the Massachusetts Board of Registration in Medicine. Our intention is to add to this list as we are made aware of additional examples or clinical scenarios.

Postimplementation Period
Prior to being launched, the system was demonstrated to all Department of Radiology modality managers, nurses, residents, fellows, and attending staff members at staff meetings, at department QA meetings, and through a series of small group and personal educational seminars. Staff members were encouraged to provide feedback and suggestions for improvement. All feedback was collected and discussed at RQMT debriefing sessions, and changes were implemented when they were considered to improve the system. The system was officially launched on October 1, 2004, and the designated departmental QA director (J.B.K.) reviewed the entered data on a daily basis. The RQMT met monthly to review all entered data and responses to the submissions once they were assigned for action and to close out cases. Summaries of these data were submitted to the departmental executive committee for review on a monthly basis. Two authors (J.B.K. and J.S.) evaluated the number of submissions into the system plotted against time (in a 9-month period) and in consensus analyzed sources of data, actions initiated, and final outcomes from October 1, 2004, to June 30, 2005. An e-mail questionnaire was sent to radiologists and residents who did not submit cases during the initial 3 months of deployment. Radiologists were given a series of options as to why they had not used the system.

Statistical Analysis
When applicable, descriptive statistics were employed, and data are presented as means ± standard errors of the mean. Differences in mean group data were considered significant at P < .05. Differences between the medians of the two unpaired groups (ie, the number of cases submitted per month prior to implementation and the number of cases submitted after implementation) were compared by using the Mann-Whitney test. Analyses were performed by using software (JMP IN 5.1; SAS Institute, Cary, NC).


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 
Number of Submissions
Over a 9-month period, there was a statistically significant increase (P < .005; Mann-Whitney test) in the number of cases submitted weekly to our online QA reporting system (mean, 21.4 cases per week) over the number of cases submitted weekly during the preceding 6 months (mean, 3.2 cases per week) (Fig 4).


Figure 4
View larger version (15K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 4: Graph shows number of case submissions before and after implementation of online QA reporting system. Data are total numbers of clinical and technical cases submitted per month. Arrow = month in which online system was deployed. Letters on x-axis indicate months, from April of one year through June of the next.

 
Among 605 total submissions, the number of clinical cases (54.4%, n = 329) far outweighed the number of technical cases (33.1%, n = 200). (The remaining cases included QA suggestions, requests and/or complaints from other departments, and cases submitted for follow-up.) In the first 4 months, clinical cases exceeded technical cases (241 total clinical cases vs 69 total technical cases); thereafter, technical case submissions increased and the ratio of clinical to technical submissions remained constant at approximately twice the number of clinical to technical submissions (Fig 4). Of all clinical submissions, only three (0.9%) were self-reports of missed diagnoses or errors.

Most clinical submissions were from the abdominal imaging section (34.4%, n = 208), followed by angiography–interventional radiology (54 cases) and ultrasonography (US) (30 cases). Most technical submissions involved the CT and magnetic resonance (MR) imaging divisions (CT, 60 cases [30.0%]; MR imaging, 42 cases [21.0%]). During the time of data acquisition, the fewest clinical submissions were received from our mammography (five cases), nuclear medicine (four cases), and neuroradiology sections (one case). Reasons why some members had not used the system varied (Table 2).


View this table:
[in this window]
[in a new window]

  		Table 2. Reasons for Lack of Usage of Online QA Reporting System

 
Submission Sources
The majority of cases (88.3%) were submitted by physicians (staff physicians, 319 cases [52.7%]; fellows, 94 cases [15.5%]; residents, 121 cases [20.0%]), with technical staff members (54 cases [8.9%]), nursing staff members (11 cases [1.8%]), and administrative staff members (six cases [1.0%]) submitting the remaining cases. Physicians reported all clinical cases, and physicians reported 176 of 200 technical cases (88.0%). Of cases reported by physicians, 37 (6.1%) were from nonradiology clinical departments; Risk Management (two cases) and our Department of Health Care Quality (three cases) also referred cases. Nonradiology physicians from our major teaching hospital and from affiliated ambulatory care facilities submitted 38 cases; 12 cases were submitted after reports from patients or their families.

Actions, Outcomes, and Categories
Of the initial 605 submissions, 149 (24.6%) were submitted for data collection, analysis, and trending, and in 106 (17.5%) of these cases, no specific or additional follow-up was considered necessary at the time of review. Of all cases submitted, 23 (3.8%) resulted in new QA and/or quality improvement projects being initiated (examples of which are provided in Table 3), and 69 (11.4%) cases resulted in individuals or entire sections undergoing additional training relating to online submission.


View this table:
[in this window]
[in a new window]

  		Table 3. Outcomes Resulting from Submissions to Online Radiology QA Reporting System

 
Clinical case categories (of 329 clinical cases) comprised 54 (16.4%) procedural complications (20 major, 34 minor), 60 (18.2%) errors in communication, 78 (23.7%) delays in diagnosis, 99 (30.1%) misses (false-negative findings), and 67 (20.4%) errors in interpretation (false-positive findings). Some cases were in more than one category. When all submissions were categorized according to type of case, 70 (11.6%) were related to patient safety; 64 (10.6%), to clinical efficiency; 85 (14.0%), to effectiveness; 250 (41.3%), to patient-centered issues; and 43 (7.1%), to the timeliness of a study. For these latter 43 cases, further analysis revealed that 23 (53%) arose from delays in reporting, while only five arose from incorrect studies being ordered. Thirty-nine cases required notification of modality managers or section chiefs, and, in 65 cases, follow-up was requested by the RQMT until the matter was resolved to our satisfaction. In terms of final outcome, 336 (55.5%) cases required no further action at the time of review (apart from the data being used for ongoing trend analysis), and nine cases were referred to our hospital-wide Quality Improvement Directors Group for review.

Twelve submissions were not related to specific cases but instead were related to quality improvement issues that were identified during study interpretation (eg, how to increase the speed of sending large files to PACS), to quality- or performance-related matters that were specific to our work (eg, the optimal Doppler US technique for evaluating transplanted liver), or to issues that arose during performance of a procedure (eg, limiting radiation exposure during CT-guided procedures).

Red Flag Data
Analysis of the red flag data revealed that 18 ED cases were uploaded after not being approved in 24 hours. After notification, 16 of these were approved within 2 hours. The 23 online ED preliminary report changes are currently being analyzed by members of both the radiology department and the ED specifically to look at the effect on patient care, the importance of the changed report, the experience of the resident reader, and to identify resident teaching opportunities arising from these errors.

Because deployment of the reporting system was a quality initiative involving the use of ongoing plan-do-study-act cycles, each addition or modification to the system was monitored and recorded (Figure 5).


Figure 5
View larger version (33K):
[in this window]
[in a new window]
[Download PPT slide]
  		Figure 5: Examples of improvements to online QA reporting system since initial deployment. Identification of these additions was made possible through continuous evaluation of the system (plan), rapid implementation of additions (do), evaluation of the implemented change (study), and continuous monitoring of the effects of these additions (act). Additions and changes were also made in response to feedback solicited from users. PDSA = plan-do-study-act.

 

   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
ADVANCES IN KNOWLEDGE
 References
 
The ultimate goal of quality assessment programs is to assess performance and to introduce feedback mechanisms that will allow the introduction of change, which in turn should lead to an improvement in the quality of care (17). Because accurate methods of data gathering are difficult to design and implement, Cascade (17) believes that what is required is the development of systematic methods, including computer databases and feedback loops. We have successfully implemented an online QA reporting system that permits us to continuously assess quality metrics, to identify and implement performance improvement tasks, and to provide feedback to all staff members who participate in the system. Compared with our previous reporting system, there has been a significant increase (P < .005; Mann-Whitney test) in the number of both clinical and technical quality-related events submitted to the online system. We do not believe that this necessarily represents an increase in the incidence of such events in our department, but rather an increase in the reporting of a number of events that were previously not reported or were dealt with in other ways.

Participation and Compliance with System
The process of implementation has not been universally well received by all of our radiologists. Not all clinical sections have fully complied with submission of clinical data. There are many possible reasons why sections or individuals might be reluctant to report errors, and we are addressing means of encouraging such data entry. One of our priorities is to facilitate data entry and workflow.

An extensive recent physician survey (18) revealed that 80% of physicians did not trust their employing hospital to run an error-reporting system. The entire issue of self reporting is controversial, but, in that survey, ensuring that submitted data were confidential and not retrievable did not satisfy those physicians who were reluctant to participate. As policies are introduced in which radiologists are expected to communicate adverse events or major errors to patients, we anticipate a further challenge to clinician participation with a reporting system such as ours.

FitzGerald (19) believes that the current culture of blame in the health care industry inhibits reporting of error and near misses. Sixteen of 56 attending radiologists (28%) and six of 36 residents (17%) in our department did not use the system during its first 4 months of being active. In response to an e-mail questionnaire attempting to analyze causes for this, 11 physicians stated that they had no reason to submit cases or that they did not know how to access the system (n = 7); others feared misuse of data or reprisal and questioned confidentiality and the possibility of discovery and protection. Many physicians believe that such systems are bureaucratic, time consuming, costly, unnecessary, and an infringement on their independence (17). Through department chiefs, our institution monitors physician performance for credentialing purposes. Only 40% of hospitals in the United States use such data for determining clinical privileges (15). Indeed, Deitch et al (15) have suggested that proper use of data for credentialing purposes requires computerized databases and commented that these did not exist in 1994. Our system has been developed in part in anticipation of future requirements.

The system currently records the name of the person submitting each case. This allows us to monitor participation and to automatically e-mail the submitter when the case is reviewed and closed. We chose to avoid establishing an anonymous reporting system to avoid the possibility of personal vendettas, realizing that this might limit reporting of clinical missed diagnoses. To further improve participation, we are considering allowing limited anonymous submissions for specific scenarios, each instance of which will be dealt with by the RQMT.

We have received few self reports of clinical errors by attending staff members. We believe that this may be due in part to most of these cases being submitted by the resident reading the case at the request of the attending physician involved in the case. Indeed, on the basis of our experience, we now encourage residents and fellows to submit cases at the time an error is identified to eliminate the possibility that attending staff members may avoid self reporting.

Benefits of Online Reporting System
Prior to the launch of this online site, each section within the Department of Radiology collected its own QA data and presented them for discussion at section QA meetings. The radiology QA director reviewed a summary of these discussions and minutes. Having all of these data presented on a screen now permits comparison of data between sections and easy identification of trends. Of the cases submitted, 11.4% have resulted in additional training of individuals, and 3.8% have resulted in initiation of specific performance improvement projects. By applying "dimensions of care" to each submission, unexpected opportunities have been identified. As an example, when a resident submitted a report describing management of a contrast material reaction, review of the case identified an opportunity for standardizing medications in our crash carts, for reevaluating and aligning our treatment of severe contrast material reactions with methods used by our cardiologists, for creating cards to be carried by all physicians describing guidelines for management of contrast material reactions, and for updating and re-presenting our annual "Management of Contrast Reactions" talk.

Educational Benefits
The Joint Commission on Accreditation of Healthcare Organizations mandates that quality improvement be included in our training curricula and that trainees participate in QA projects. Our online system facilitates identification of potential performance improvement projects for residents during their quality improvement electives. A recent example resulted from a radiology resident submission questioning appropriate utilization of CT studies by ED residents. This allowed us to assign the case as a resident elective project and has resulted in an educational process teaching ED residents how to order the most appropriate imaging studies. Because a feedback loop is included in the submission process, all participants receive feedback on the outcome of each of their submissions once it has been reviewed and closed. In this way, an educational component is included in which members of the department actually get to see exactly how their quality concerns can be translated directly into improved performance in their workplace.

An example submitted to us illustrates the positive educational and clinical benefits of the system. An abdominal imaging fellow reported a lack of consistency when US technologists scanned the livers of patients after transplantation. Our response was first to review the protocol to measure how compliant our sonographers were with departmental scanning guidelines and the protocol for patients with liver transplants. A prospective 3-month survey was performed and revealed that certain US technologists required additional training to improve their compliance with the protocols. This also resulted in an opportunity to upgrade the scanning protocols, along with a seminar on Doppler US for liver transplantation. The system has thus provided unexpected educational benefits, and more and more technologists and residents are realizing that reporting of seemingly minor problems may have a positive educational benefit for them and their colleagues. We believe that this is one reason the number of submitted cases continues to grow on a weekly basis.

Performance Improvement
The system facilitates identification of opportunities for improvement. Residents in our hospital are offered quality improvement electives. Having reviewed examples of cases submitted to our online system, one resident chose to study ways of reducing errors in communication between radiology and other hospital departments. A review of the database and a search of the "communication error" data section revealed that errors were occurring owing to vaguely defined criteria and mechanisms for communication of critical radiology test results. This resident chose to focus on these areas, and, together with the RQMT and with input from the Massachusetts Coalition of Hospital Safety, the resident defined a set of clinical criteria and findings, such as unexpected pneumothorax, that required urgent communication with the ordering physician, along with a set of conditions that required less urgent (<3 days) direct communication. Together with these clinical conditions, a new policy for communication of critical radiologic findings was written and approved for our department and now serves to guide our interpretation and communication of abnormal results.

Enhanced Function of Reporting System
In response to user feedback, new changes have been implemented and many improvements have been made to the system since it was initially deployed. As an example, we have added a search engine not only for data analysis but also for selection of data that are transmitted to section chiefs and modality managers. Prior to section- or department-wide QA meetings, specific data can be extracted for analysis and presentation. The best example has been an accreditation site visit requiring specific US QA data to be made available to the investigator. A rapid extraction of all technical and clinical US quality cases provided useful data for presentation. A second improvement has been the incorporation of an automatic e-mail system for reminding reviewers that their cases are due for response (when no response has occurred after 2 weeks). This automatic e-mail function also notifies members of the RQMT that red flag cases have been received.

Limitations
We identified several limitations to this system, and have attempted to minimize these. Limitations were continuously sought as part of a plan-do-study-act feedback cycle. First, staff resistance to participation was due in part to concern for data security and privacy and risks associated with external review or discovery. We were assured through our Risk Management Department that all data were secure and protected, and we communicated this fact to all staff. In addition, in response to a submitted non–case-related question and request, this topic was covered at a department-wide QA meeting. A more challenging limitation to data submission is self implication and prevention of personal blame. The site certainly permits cases to be submitted that may appear to target specific members of the department or sections, but, in our experience, this issue was easily dealt with through rational peer review discussion. Other limitations we identified included lack of repeat submissions when staff members perceived that their prior submissions did not effect any meaningful results. For this reason, once a case is closed, an e-mail is now sent to the submitter inviting him or her to review the findings and outcome of the submission. Also, the time and thus (indirectly) the money required for data management and analysis should not be underestimated. Effective management of such a system requires a dedicated, interested, and enthusiastic team of qualified members.

When analyzing and presenting accumulated clinical data, it is important to compare the number of submissions with work volume. As an example, procedural complications reported must be related to the overall number of procedures performed and to case complexity. Similarly, missed CT diagnoses must ultimately be compared with a denominator of total interpreted cases over time. In this way, presented data would reflect performance more accurately.

Future Plans
In an ongoing effort to facilitate and encourage data entry, we have recently started developing a personal digital assistant (PDA) version of the online reporting system. This will allow staff members to input data directly from their PDAs. In addition, we have recently provided access to QA directors from other departments in our hospital. This will allow them to directly input data into the system and allow them to track action items and resolution of all submitted cases. It is our hope that this will permit us to submit similar QA issues to non-radiology departments. As an extension of this plan, the software has been specifically written so as to be easily modified and thus usable by other departments. The long-term plan is that this online QA reporting system will be deployed on a hospital-wide basis, and work toward achieving this goal has begun. We have already included all off-campus radiology outpatient offices and affiliated community hospitals and anticipate no difficulties with additional sites or departments using this system.

In summary, in response to the need for improved QA reporting, we have implemented an on-line radiology QA reporting system in our department. A secure online QA reporting system encourages ongoing submission of quality data, serves as a database for identifying and managing trends, and can be utilized for educational, maintenance, and clinical decision making.


   ADVANCES IN KNOWLEDGE
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
References
 


   FOOTNOTES
 

Abbreviations: ED = emergency department • PACS = picture archiving and communication system • QA = quality assurance • RQMT = radiology quality management team

Authors stated no financial relationship to disclose.

Author contributions: Guarantor of integrity of entire study, J.B.K.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, J.B.K., C.S.Y.; clinical studies, J.B.K., C.S.Y., J.S., D.T.H., Y.J.M.; experimental studies, J.B.K., C.S.Y., H.Y.K.; statistical analysis, J.B.K., C.S.Y.; and manuscript editing, all authors


   References
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 ADVANCES IN KNOWLEDGE
 References
 

  1. Thrall JH. Quality and safety revolution in health care. Radiology 2004;233:3–6.[Free Full Text]
  2. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I. N Engl J Med 1991;324:370–376.[Abstract]
  3. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients: results of the Harvard Medical Practice Study II. N Engl J Med 1991;324:377–384.[Abstract]
  4. Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA 1995;274:35–43.
  5. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events: implications for prevention. ADE Prevention Study Group. JAMA 1995;274:29–34.
  6. Chassin MR, Galvin RW. The urgent need to improve health care quality: Institute of Medicine National Roundtable on Health Care Quality. JAMA 1998;280:1000–1005.[Abstract/Free Full Text]
  7. Cullen DJ, Sweitzer BJ, Bates DW, et al. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med 1997;25:1289–1297.[CrossRef][Medline]
  8. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38:261–271.[CrossRef][Medline]
  9. Donnelly LF, Strife JL. Performance-based assessment of radiology faculty: a practical plan to promote improvement and meet JCAHO standards. AJR Am J Roentgenol 2005;184:1398–1401.[Abstract/Free Full Text]
  10. Soffa DJ, Lewis RS, Sunshine JH, Bhargavan M. Disagreement in interpretation: a method for the development of benchmarks for quality assurance in imaging. J Am Coll Radiol 2004;1:212–217.[CrossRef][Medline]
  11. Leape LL, Cullen DJ, Clapp MD, et al. Pharmacist participation on physician rounds and adverse drug events in the intensive care unit. JAMA 1999;282:267–270.[Abstract/Free Full Text]
  12. Cullen DJ, Bates DW, Leape LL; Adverse Drug Event Prevention Study Group. Prevention of adverse drug events: a decade of progress in patient safety. J Clin Anesth 2000;12:600–614.[CrossRef][Medline]
  13. Kohn LT, Corrigan JM, Donaldson MS, eds. Why do errors happen? In: To err is human: building a safer health system. Washington, DC: National Academy Press, 2001; 49–68.
  14. Institute of Medicine. Using information technology. In: Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press, 2001; 164–180.
  15. Deitch CH, Chan WC, Sunshine JH, Zinniger MD, Cascade PN, Cochran ST. Quality assessment and improvement: what radiologists do and think. AJR Am J Roentgenol 1994;163:1245–1254.[Abstract/Free Full Text]
  16. Renfrew DL, Franken EA, Berbaum KS, Weigelt FH, Abu-Yousef MM. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992;183:145–150.[Abstract]
  17. Cascade PN. Quality improvement in diagnostic radiology. AJR Am J Roentgenol 1990;154:1117–1120.[Free Full Text]
  18. White C. Doctors mistrust systems for reporting medical mistakes. BMJ 2004;329:12–13.[Free Full Text]
  19. FitzGerald R. Radiological error: analysis, standard setting, targeted instruction and teamworking. Eur Radiol 2005;15(8):1760–1767.[CrossRef][Medline]




This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow Full Text (PDF)
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Kruskal, J. B.
Right arrow Articles by Kressel, H. Y.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Kruskal, J. B.
Right arrow Articles by Kressel, H. Y.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
RADIOLOGY RADIOGRAPHICS RSNA JOURNALS ONLINE