|
|
||||||||
Health Policy and Practice |
1 From the Radiology Management Group, Department of Radiology, Brigham & Womens Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115 (S.O.P., J.G.B., K.H.Z., A.G., L.A.I., P.K., S.E.S., P.R.R.); Department of Health Policy and Management, Harvard School of Public Health, Boston, Mass (S.O.P.); and Department of Health Care Policy, Harvard Medical School, Boston, Mass (K.H.Z.). From the 2002 RSNA scientific assembly. Received July 22, 2003; revision requested October 3; final revision received February 20, 2004; accepted March 26. Supported in part by the Agency for Healthcare Research and Quality of the National Institutes of Health HS1323401. Address correspondence to S.O.P. (e-mail: sondateguiparra@partners.org).
| ABSTRACT |
|---|
|
|
|---|
MATERIALS AND METHODS: This investigation met the criteria for an exemption from institutional review board approval. A cross-sectional study in which a validated national survey was sent to members of the Society of Chairmen of Academic Radiology Departments (SCARD) was conducted. The survey was designed to examine the following six categories of 28 performance indicators: (a) general organization, (b) volume and productivity, (c) radiology reporting, (d) access to examinations, (e) customer satisfaction, and (f) finance. A total of 158 variables were included in the analysis. Summary statistics, the
2 test, rank correlation, multiple regression analysis, and analysis of variance were used.
RESULTS: A response rate of 42% (55 of 132 SCARD members) was achieved. The mean number of performance indicators used by radiology departments was 16 ± 6.35 (standard deviation). The most frequently utilized performance indicators were as follows: (a) productivity, in terms of examination volume (78% [43 departments]) and examination volume per modality (78% [43 departments]); (b) reporting, in terms of report turnaround (82% [45 departments]) and transcription time (71% [39 departments]); (c) access, in terms of appointment access to magnetic resonance imaging (80% [44 departments]); (d) satisfaction, in terms of number of patient complaints (84% [46 departments]); and (e) finance, in terms of expenses (67% [37 departments]). Regression analysis revealed that the numbers of performance indicators in each category were statistically significant in predicting the total number of performance indicators used (P < .001 for all). Numbers of productivity and financial indicators were moderately correlated (r = 0.51). However, there were no statistically significant correlations between the numbers of performance indicators used and hospital location, hospital size, or department size (P > .4 for all).
CONCLUSION: Assessing departmental performance with a wide range of management indicators is not yet an established and standardized practice in academic radiology departments in the United States. Among all indicators, productivity indicators are the most frequently used.
© RSNA, 2004
Index terms: Economics, medical Radiology and radiologists, departmental management Radiology and radiologists, socioeconomic issues
| INTRODUCTION |
|---|
|
|
|---|
Traditionally, health care management has been equated with financial management. It has been suggested that in addition to financial outcomes, health care organizations should assess intangible assets that affect the bottom line such as clinical processes, staff skills, and patient satisfaction (3).
The field of radiology is an essential component of clinical medicine, and it will continue to grow in the future (4). Driven by the external scrutiny of regulatory and accreditation agencies, as well as by the internal desire to improve the quality of care while lowering cost, interest in quantitative assessment of various aspects of radiology department performance has grown dramatically (59). In radiologic practice, several of the key departmental systems lend themselves to the use of simple, objective, numerical monitoring data. However, to our knowledge, there are no published reports of an appropriate set of performance indicators that can be used by academic or nonacademic radiology departments. The key performance indicators that reflect the mission, vision, and strategic direction of an organization are collectively referred to as "dashboard indicators" (1012). Such a dashboard of performance indicators can be used to monitor and navigate the organization in the same way that a pilot uses the cockpit dashboard to navigate a plane. In radiology, the dashboard of performance indicators can be depicted graphically to provide a comprehensive snapshot of all ongoing departmental activities.
The purpose of the present study was to determine the management performance indicators most frequently utilized in academic radiology departments in the United States.
| MATERIALS AND METHODS |
|---|
|
|
|---|
Survey
A cross-sectional multi-institutional survey study was conducted among academic radiology departments across the country from May to November 2002. An electronic survey questionnaire was designed after an extensive literature review and was pilot tested in four teaching hospitals. Prior to pilot testing, approval was obtained from the president of the Society of Chairmen of Academic Radiology Departments (SCARD). On the basis of responses obtained from the pilot study, the questionnaire was modified and the final survey was sent across the nation to all 132 members of SCARD. Face validity during pilot testing was established by focus groups and expert panels. Pilot testing focused on improving the format and contents of the questionnaire by reducing its length and increasing the clarity of questions. To increase the response rate, the questionnaire was sent in stages up to three times to all members by e-mail, once every 3 weeks, and by fax 2 months after the last electronic version. The responses were kept confidential and anonymous in that every hospital was assigned an identification number by the Web master. The process was automated, and questionnaires were automatically sent again to nonresponding hospitals. The investigators had access only to the final database with identification numbers. The electronic questionnaire was designed so that the responder could not move forward to the next question without answering the previous one. Thus, all questionnaires that were returned were completely answered.
Six categories of 28 performance indicators were examined. A more detailed definition of the study terms is provided in the Appendix. A total of 158 variables were included in the analysis. The categories were as follows: (a) general organization (11 variables), (b) volume and productivity (nine indicators, 50 variables), (c) radiology reporting (three indicators, 18 variables), (d) access to examinations (four indicators, 30 variables), (e) customer satisfaction (five indicators, 28 variables), and (f) finance (seven indicators, 21 variables). The survey questionnaire was designed on the basis of the most common performance indicators used in health care settings, according to an extensive review of the literature (23,1316). Information obtained for each category of performance indicators consisted of (a) which indicators the department used and (b) how often they were monitored.
General Organization
The general organizational indicators included the following: (a) the region of the country in which the hospital was located (Pacific, Southwest, Midwest, Northeast or South), (b) the number of beds at the hospital, (c) the number of radiologic examinations performed per year, and (d) the number of full-time employees working in the department. We used the number of beds and the number of radiologic examinations as proxies for the size of the hospital and the size of the radiology department, respectively.
Productivity
The following productivity indicators were studied: (a) examination volume, (b) examination volume per modality, (c) technical relative value units (RVUs), (d) professional RVUs, (e) technical RVUs per full-timeequivalent (FTE) employee, (f) professional RVUs per FTE employee, (g) gross charges by modality, (h) collections by FTE employee, and (i) volume by resource or device.
Radiology Reporting
The third category of performance indicators studied was radiology reporting. This included questions regarding transcription time (time from verbal dictation of examination results to preliminary results), signature time (time from preliminary results to report finalization), and report turnaround time (time from examination completion to report finalization).
Access to Examinations
The next set of performance indicators surveyed was regarding patient access to examinations, both on an inpatient and on an outpatient basis. The aim was to study whether the institutions monitored patient waiting time to get an appointment for the following examination modalities: (a) magnetic resonance (MR) imaging, (b) mammography, (c) nuclear imaging, and (d) computed tomography (CT).
Customer Satisfaction
"Customers" in the field of radiology comprise patients and their referring physicians, as well as other radiology staff members (15). The customer satisfaction indicators that were studied included the following: (a) patient satisfaction, (b) ambulatory waiting time, (c) patient complaints, (d) referring physician satisfaction, and (e) employee satisfaction.
Finance
Finally, the survey included financial indicators such as (a) cost per RVU, (b) average RVU per examination, (c) hours worked per RVU, (d) collections by modality, (e) accounts receivable, (f) supply cost per RVU, and (g) expenses.
Statistical Methods
Statistical methods included summary statistics, the Pearson
2 test of independence, nonparametric Spearman rank correlation analysis, multiple regression analysis, and analysis of variance (ANOVA). First, a descriptive analysis was performed to assess the general organizational characteristics, as well as the number, type, and monitoring frequency of performance indicators used by radiology departments. We also analyzed the general organizational characteristics of the departments by using the minimum and maximum numbers of performance indicators, as well as the most frequently monitored indicators in each category.
Second, we employed the Pearson
2 test to analyze whether there was any relationship between the type of performance indicators used and the U.S. location of the hospital, its size, or the number of radiologic examinations performed per year.
Third, nonparametric correlation analysis with the Spearman rank correlation coefficient (r) was performed to assess the following: whether larger hospitals used more performance indicators, whether there was a correlation between the size of the hospital and the number of radiologic examinations performed annually, whether the total number of performance indicators used varied according to the number of radiologic examinations performed, and whether there was a correlation between each category of performance indicators. We also evaluated whether there was any association between the number of FTE employees and the performance indicators utilized by the departments. Statistical tests were used to assess whether r was greater than 0, with P values reported.
Fourth, we used a multiple regression analysis to assess whether the total number of performance indicators (outcome as the dependent variable) adopted by the departments was determined by the U.S. region where the hospital was located, the size of both the hospital and the radiology department, and the number of performance indicators used in each of the six established categories (explanatory or independent variables). Only the variables that were statistically significant (P < .05) in the bivariate analysis were reported in the final model.
Finally, using ANOVA, we assessed whether there was a significant difference in the number of performance indicators per category or the total number of performance indicators used according to the regional location or the size of the hospital.
All statistical analyses described above were performed by using the software program SPSS 11 (17). All analyses were performed by three authors in consensus.
| RESULTS |
|---|
|
|
|---|
Geographic Region
Our analysis revealed that of the 55 responding hospitals, 20 (36%) were in the Northeast region and 18 (33%) were in the Midwest region. About five hospitals (9%) were in the Southwest region, while four (7%) were in the Pacific region. Nearly 15% (ie, eight) of all the responding hospitals were in the South region.
Size
Thirty-two (about 58%) of the 55 responding hospitals had more than 500 beds, while 21 (38%) had between 200 and 500 beds in operation. Only two (4%) of the responding hospitals had fewer than 200 beds in operation. Twenty-eight (approximately 51%) of the departments performed between 200 000 and 400 000 examinations per year. Seventeen (nearly 31%) of the departments performed fewer than 200 000 examinations per year, whereas 10 (18%) performed more than 400 000 examinations per year. The mean number of FTE employees working in the departments was 146.81 ± 202.96 (standard deviation).
Number of Performance Indicators
The total number of performance indicators used ranged from 0 to 28. Of the 55 responding departments, three (5%) used fewer than six performance indicators, while four (7%) used more than 24 performance indicators. The four departments that used fewer than six performance indicators were from the Midwest and Northeast regions, had more than 200 beds in operation, and performed more than 200 000 examinations annually. Similarly, those that used more than 24 performance indicators belonged to the Southwest, Midwest, and Northeast regions. Three of these four departments had more than 500 beds in operation, while one had between 200 and 500 beds in operation. All departments performed more than 200 000 examinations annually. Furthermore, the departments used a mean of 16 performance indicators ± 6.35, and the productivity indicators were used most frequently: Of these 16 performance indicators, five were productivity indicators. Only two were related to radiology reporting, while three each were related to access to examinations, customer satisfaction, and financial outcomes (Fig 1). Fourteen (25%) of the 55 hospitals used 10 or fewer indicators to monitor the performance of the radiology department.
|
|
|
|
|
|
|
|
2 testing showed that in the access-to-examinations category, use of the access to MR imaging indicator was significantly associated with the size of the radiology department (P = .048). All other types of performance indicators did not vary significantly according to the location of the hospital or according to the size of both the hospital and the radiology department (P > .07 for all). In addition, ANOVA revealed that the number of performance indicators used in each category and the total number of performance indicators used by the institutions surveyed did not differ significantly according to the location or the size of the hospital.
Spearman Rank Correlation Analysis
The correlation analysis based on the Spearman rank correlation coefficient (r) revealed that the number of radiologic examinations performed per year was moderately correlated with the size of the hospital (r = 0.53, P < .001). The numbers of productivity and financial indicators were also moderately correlated (r = 0.51, P < .001). The numbers of productivity indicators were minimally correlated with the numbers of reporting indicators (r = 0.31, P = .02) and the numbers of access indicators used (r = 0.29, P = .03). Additionally, the numbers of report indicators used were minimally correlated with the numbers of access indicators used (r = 0.29, P = .03) and the numbers of financial indicators used (r = 0.34, P = .01). Finally, the numbers of customer satisfaction indicators used were minimally correlated with the numbers of financial indicators used (r = 0.31, P = .02). The analysis also revealed that there was a minimal correlation (r = 0.27, P = .04) between the number of radiologists and the number of financial indicators adopted. However, there was no statistically significant correlation between the number of FTE employees and the total numbers of indicators used or the numbers of indicators used in other categories (P > .07 for all). Furthermore, the total number of performance indicators used did not vary according to the size of the hospital or the radiology department.
Multiple Regression Analysis
Regression analysis revealed that neither the region in which the hospital was located (P = .98) nor the size of the hospital (P = .86) nor the size of the radiology department (P = .239) was significant in predicting the set of performance indicators used by each department. However, the numbers of performance indicators in each category (customer satisfaction, access to examinations, productivity, and radiology reporting) were statistically significant in predicting the total number of performance indicators used (P < .001 for all).
| DISCUSSION |
|---|
|
|
|---|
Our survey results showed that about 95% of responding academic radiology departments in the United States measure their performance. However, there was little consistency in the types of performance indicators utilized or in the amount or the frequency of monitoring. In addition, the mix of performance indicators used was limited to a few categories.
According to our analysis, the majority of the performance indicators used belonged to the category of productivity, with examination volume being the most prevalent in that category. Furthermore, our correlation analysis results showed that those institutions that used greater numbers of productivity indicators also used more financial indicators. This correlation is reasonable because productivity and financial indicators are easily quantifiable and objectively measured. As a result, they are most frequently adopted by departments to assess performance. Contrary to the assumption that larger hospitals would use a large number of indicators to monitor their performance, our results showed that the total number of performance indicators used did not correlate with either the size of the hospital or the size of the radiology department.
A key finding of our analysis is the lack of consistency in the metrics adopted by all departments. Uniformity in measurement will not only allow a meaningful assessment of performance trends over time within a particular department but will also facilitate valid comparisons across departments nationally. This is essential for establishing minimum levels of performance (understood as a standard). Thus, the development of a dashboard to measure performance indicators uniformly in radiology is a relevant future application of the study. A dashboard gives a systemic view of the functioning of the department from the point of first contact to patient discharge. It facilitates faster understanding of trends, problems, and opportunities. The results of this study reveal the need for radiology departments to develop such a dashboard.
Some of the potential limitations of our study were as follows: (a) bias in the selection of indicators for the questionnaire, (b) inclusion of only academic radiology departments in the survey, (c) lack of adjustment for multiple testing, and (d) the response rate. There was no unanimously accepted set of indicators for performance measurement in radiology published in the literature. The performance indicators included in our survey questionnaire were those that were adopted by radiology departments in four large hospitals affiliated with medical schools (as seen during pilot testing) and those that are used most frequently in health care settings (23,1316).
However, there may be other performance indicators that are monitored by other radiology departments that were not included in the survey. For example, in an academic center, teaching and research are as important as clinical productivity. However, in this survey, we were focusing on clinical productivity metrics. Teaching and research productivity metrics were beyond the scope of this study, and, hence, our questionnaire did not assess indicators used to measure teaching and research productivity that may be adopted by some departments.
Similarly, comparing performance indicators used in academic and private settings was beyond the scope of our study. Thus, the performance indicators assessed in this survey may not be exhaustive; furthermore, they may be less relevant for private settings. We would have to take this into account before extending these results to nonacademic settings.
Although we performed a number of statistical tests, we did not adjust for multiple comparisons because we did not perform all pair-wise comparisons over "ordinal" categories in the survey. Instead, inferences were made mainly by using nonparametric Spearman correlation coefficients with P values reported. Finally, the response rate obtained in our study was 42%. Although a response rate of 70% or greater is considered to be optimal (20), literature research has shown that response rates are typically greater than 20% (2123). Other studies have also shown that response rates to survey questionnaires can vary from 27% to 90% (6,21,24). A potential limitation of our response rate is that academic centers without performance indicators would be less likely to respond to the survey and thus may be underrepresented in our results. This would tend to upwardly bias the frequency of use of performance indicators.
In summary, the results of our study indicate that academic radiology departments in the United States do not use a comprehensive set of indicators to monitor performance, and there is no consistency among departments as to which indicators are used. On the average, the majority of the performance indicators that are used monitor productivity. Further research is necessary to develop a dashboard to assess performance in these departments.
| APPENDIX |
|---|
|
|
|---|
Benchmark: comparative standard against which others may be compared. The value is usually calculated by using academic specialtyspecific billing data to determine statistical comparisons.
Full-time equivalent: the percentage of full time a provider spends in billable clinical activity.
Management performance indicators: indicators utilized for management of departmental activities to improve performance.
Relative value unit: a nonmonetary unit of measure used to express the time, complexity, and cost of performing a given service relative to those of performing other procedures.
Variable: a characteristic that can be measured or categorized quantitatively.
| FOOTNOTES |
|---|
Authors stated no financial relationship to disclose.
Author contributions: Guarantors of integrity of entire study, S.O.P., P.R.R.; study concepts, S.O.P., P.R.R.; study design, S.O.P., P.R.R., K.H.Z., P.K.; literature research, S.O.P., J.G.B., A.G., P.R.R.; data acquisition, S.O.P., S.E.S., P.R.R.; data analysis/interpretation, S.O.P., J.G.B., K.H.Z., A.G., L.A.I.; statistical analysis, S.O.P., J.G.B., K.H.Z.; manuscript preparation, J.G.B., S.O.P., K.H.Z., P.R.R., L.A.I.; manuscript definition of intellectual content, S.O.P., P.R.R., S.E.S.; manuscript editing, J.G.B., S.O.P., K.H.Z., L.A.I., A.G., P.K., S.E.S.; manuscript revision/review and final version approval, all authors
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
S. Ondategui-Parra, S. M. Erturk, and P. R. Ros Survey of the use of quality indicators in academic radiology departments. Am. J. Roentgenol., November 1, 2006; 187(5): W451 - W455. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Ondategui-Parra, J. G. Bhagwat, K. H. Zou, E. Nathanson, I. E. Gill, and P. R. Ros Use of Productivity and Financial Indicators for Monitoring Performance in Academic Radiology Departments: U.S. Nationwide Survey Radiology, July 1, 2005; 236(1): 214 - 219. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |