HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) All Versions of this Article: 2473071431v1 247/3/771    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Lockyer, J. M. Articles by Fidler, H. M. Search for Related Content PubMed PubMed Citation Articles by Lockyer, J. M. Articles by Fidler, H. M.

Assessment of Radiology Physicians by a Regulatory Authority¹

¹ From the Office of Continuing Medical Education, Faculty of Medicine, University of Calgary, 3330 Hospital Drive NW, Calgary, Alberta T2N 4N1, Canada. Received August 10, 2007; revision requested October 15; revision received December 12; accepted January 10, 2008; final version accepted January 12. Supported in part by the College of Physicians and Surgeons of Alberta. Address correspondence to J.M.L. (e-mail: lockyer{at}ucalgary.ca).

	ABSTRACT

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 
Purpose: To determine whether it is possible to develop a feasible, valid, and reliable multisource feedback program for radiologists.

Materials and Methods: Surveys with 38, 29, and 20 items were developed to assess individual radiologists by eight radiologic colleagues (peers), eight referring physicians, and eight co-workers (eg, technicians), respectively, by using five-point scales along with an "unable to assess" category. Radiologists completed a self-assessment on the basis of the peer questionnaire. Items addressed key competencies related to clinical competence, collegiality, professionalism, workplace behavior, and self-management. The study was approved by the University of Calgary Conjoint Health Ethics Research Board.

Results: Data from 190 radiologists were available. The mean numbers of respondents per physician were 7.5 of eight (1259 of 1520, 83%), 7.15 of eight (1337 of 1520, 88%), and 7.5 of eight (1420 of 1520, 93%) for peers, referring physicians, and co-workers, respectively. The internal consistency reliability indicated all instruments had a Cronbach of more than 0.95. The generalizability coefficient analysis indicated that the peer, referring physicians, and co-worker instruments achieved a generalizability coefficient of 0.88, 0.79, and 0.87, respectively. The factor analysis indicated that four factors on the colleague questionnaire accounted for 70% of the total variance: clinical competence, collegiality, professional development, and workplace behavior. For the referring physician survey, three factors accounted for 64.1% of the variance: professional development, professional consultation, and professional responsibility. Two factors on the co-worker questionnaire accounted for 63.2% of the total variance: professional responsibility and patient interaction.

Conclusion: The psychometric examination of the data suggests that the instruments developed to assess radiologists are a feasible way to assess radiology practice and provide evidence for validity and reliability.

© RSNA, 2008

	INTRODUCTION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 
In the United States and in Canada, the Accreditation Council for Graduate Medical Education (ACGME) competencies (1) and the Royal College of Physicians and Surgeons of Canada (CanMEDS, formerly Canadian Medical Education Directions for Specialists) roles (2) have established key parameters for performance in practice. A new emphasis on interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice, along with medical knowledge and patient care, is redirecting both residency and maintenance of certification initiatives (3). This new emphasis is also stimulating discussion about practice performance assessment that provides performance feedback, improves work flow, and improves practice efficiency (3). Radiologic colleague review, patient, and referring physician surveys are being considered as potentially valuable mechanisms for physician performance assessment. A recent publication profiled the feasibility of scorecards to promote professionalism and effective communication (4). Radiologic colleague (peer) review of diagnostic imaging has been advocated for error reduction and improved patient care (5).

Multisource feedback (MSF), sometimes termed 360-degree feedback, has been used for the assessment of ACGME core competencies for radiology residents (6). In MSF, aggregate data collected through questionnaires from medical colleagues, referring physicians, co-workers, and patients provides feedback about behaviors that can be addressed and changed. This form of evaluation is currently used as part of revalidation and quality improvement for practicing physicians in some specialties such as internal medicine (7–9), pediatrics (9,10), and emergency medicine (11). MSF can be designed to elicit feedback about all of the competencies.

Studies of MSF show that reliable and valid instruments (questionnaires) can be developed (6–12). It appears feasible to develop quality improvement programs in which most of the physicians in the discipline can be assessed by eight to 10 co-workers, eight to 10 medical colleagues, and 25 patients (6–12). This number of raters produces an acceptable reliability for both the overall instrument and the physician being assessed (6–12). Furthermore, given that the intent of MSF is to guide professional development, studies have shown that participating physicians will use their feedback data to guide the changes they make (7,13).

The main purpose of our study was to conduct an MSF study of practicing radiologists by using data provided by peers, referring physicians, co-workers, and self to assess the feasibility, validity, and reliability of an MSF system. While these aspects of an MSF program show promise for residents (6), a large-scale study involving practicing radiologists has not been reported. Our study had several focused questions: (a) What is the feasibility of an assessment system for radiologists that provides feedback from peers, referring physicians, co-workers, and self? (b) What questions about a radiologist's practice can assessors answer? (c) What are the score profiles for each of the items (ie, mean and standard deviation) on the surveys? (d) Do the items on a survey group combine into meaningful scales to guide performance improvement direction? (e) Are the instruments reliable for the practice of radiology and for the individual physician who is assessed?

	MATERIALS AND METHODS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 
The College of Physicians and Surgeons of Alberta–Physician Achievement Review program began developing MSF instruments in 1996 (14). This program is mandatory and requires that every physician participate on a 5-year cycle. The original goal of the program was to provide feedback to physicians regarding six broad categories of performance: medical knowledge and skills, attitudes and behavior, professional responsibilities, practice improvement activities, administrative skills, and personal health. Instruments have been developed and tested for several disciplines (9–11,14). As part of this work, a set of instruments was created and psychometrically tested for radiologists. The researchers received approval from the University of Calgary's Conjoint Health Research Ethics Board to undertake a psychometric analysis of the data collected. No identifying data (eg, sex, age, year of graduation from medical school, or location of school) were provided in the data set.

The instruments (questionnaires) were developed by a working group of radiologists. The committee drew on previous instruments (14) and the CanMEDS competencies (2) to develop the items for each of the four questionnaires (peer, referring physician, co-worker, and self). The goal for each survey was to have a reasonable number of items to provide feedback regarding most aspects of practice across the three surveys. Unlike other Physician Achievement Review instruments, there was no patient component. Having two instruments for physician colleagues (peer and referring physician) recognized that the feedback provided regarding practice would be optimized by having two sources of input, each with a discrete set of questions. After the committee had developed the questionnaires, they were mailed to every radiologist in the province for feedback. Questionnaires were modified following that feedback. The final set of questionnaires is provided in Tables 1–4.

Each physician was responsible for completing a self-assessment and identifying the eight peers, eight referring physicians, and eight co-workers who could answer the questions on the survey. Previous work had established that raters chosen by people being assessed do not provide significantly different evaluations than those selected by a third party (8). Each radiologist was asked to provide eight assessors for each survey because previous studies (7,8,10–12) have shown that 8–12 surveys are likely to produce a generalizability coefficient of 0.70 or higher, suggesting that the data provided to each physician are reliable (stable).

Pivotal Research (Edmonton, Canada), a private company that handles the College of Physicians and Surgeons of Alberta–Physician Achievement Review program, recruited the physicians under the direction of the College of Physicians and Surgeons of Alberta. These physicians had to be licensed to practice and have been in practice in the province for at least 3 years. Pivotal Research provided the assessors with copies of the questionnaire and responses went directly to Pivotal Research. Pivotal Research initiated reviews with all 241 radiologists who met the eligibility criteria between May and September 2006. Participation in the Physician Achievement Review program is mandatory for all physicians in the province, although physicians can be exempt or their participation deferred because they are no longer practicing in the province, have retired, have been in their current location for less than 3 years, or provide evidence of extenuating personal circumstances.

A number of statistical analyses were undertaken to address the research questions posed. Response rates were used to determine feasibility for each of the respondent groups (question 1). For each item on each survey, the percentage of "unable to assess" data, as well as the means and standard deviations, were computed to determine the viability of items and the score profiles (for questions 2 and 3, respectively). Items where "unable to assess" data exceeds 15% on a survey may be in need of revision or deletion.

We used exploratory factor analysis to determine which items on each survey belonged together (ie, became a factor or scale) (research question 4). This analysis allowed us to identify the factors and numbers of factors for each instrument and describe the relative variance accounted for by using each factor and its coherence. In this study, by using individual physician data as the unit of analysis for each survey, the items were intercorrelated by using Pearson product-moment correlations. The correlation matrix was then separated into principal components, which were subsequently rotated to the normalized varimax calculation criterion. Items were considered to be part of a factor if their primary loading was on that factor. The number of factors to be extracted was based on the Kaiser rule (ie, Eigen values of >1.0).

The factors or scales established through exploratory factor analysis were then used to establish the key domains (eg, clinical competence) for improvement, while the items in each factor would provide more precise information about specific behaviors (eg, works at a reasonable pace, works to resolve conflict in the workplace, has effective verbal communication skills). Physician improvement could be guided by scores on factors or items.

Reliability was assessed (research question 5). Internal consistency reliability was examined by using the Cronbach coefficient for each of the rater groups and for each of the scales and/or factors for each rater group. This enabled an assessment of overall instrument internal consistency. This analysis was followed by a generalizability analysis to determine the generalizability coefficient to ensure that there were sufficient numbers of items and raters to provide stable data for each radiologist on each instrument. Normally, a generalizability coefficient of 0.70 or higher suggests that data are stable (7–11). If the generalizability coefficient is low, it suggests that more raters or more items would be required to enhance stability.

	RESULTS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 
We analyzed data for 190 of 241 radiologists. These physicians all provided self-assessment data and their data sets were considered complete by January 2007, when the data collection phase of the study was closed. Of the 241 reviews initiated, 27 were deferred or exempt. Data from another 24 physicians were in process (ie, awaiting responses) and not made available for analysis. Data related to the physician's sociodemographic background were not provided.

A total of 1259 (83%) of 1520 peer questionnaires, 1337 (88%) of 1520 referring physician questionnaires, and 1420 (93%) of 1520 co-worker questionnaires were available on the basis of eight surveys per source per physician.

The majority of items on the questionnaires could be answered by respondents. As presented in Tables 1, 3, and 4, review of "unable to assess" items showed that eight (of 29) items on the referring physician survey and one item (of 20) on the co-worker questionnaire had "unable to assess" rates of 15% or higher. There were no items on the peer questionnaire with "unable to assess" rates of 15% or higher. With the exception of one item on the self-assessment questionnaire, all items had mean measurements between 4 and 5 on the five-point scale. The means on the self-assessment were lower than the means on the peer questionnaire.

The factor analysis showed that the data on the peer questionnaire separated into four factors that accounted for 70% of the total variance: clinical competence, collegiality, professional development, and workplace behavior (Table 2). For the referring physician survey, three factors accounted for 64.1% of the variance: professional development, professional consultation and professional responsibility (Table 3). In the co-worker questionnaire, two factors accounted for 63.2% of the variance: professional responsibility and patient interaction (Table 4).

Reliability analyses (Cronbach of internal consistency reliability) indicated that all of the instruments' full scales had high internal consistency reliability (Cronbach > 0.95). The reliability for the factors (subscales) in each questionnaire had high internal consistency reliability (Cronbach > 0.85). The generalizability coefficient analysis showed that the peer, referring physician, and co-worker instruments resulted in a generalizability coefficient of 0.88, 0.79, and 0.87, respectively.

	DISCUSSION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 
In this study, we developed and evaluated an MSF set of questionnaires to assess radiologists by using their peers, referring physicians, and co-workers. The analysis shows evidence for the feasibility of the process and the reliability and validity of the instruments. Physicians were assessed on a number of aspects of practice that the regulatory authority and the physicians themselves (through their participation on the committee and feedback about the questionnaires) believed to be important. While not designed to specifically assess ACGME (1) or CanMEDS (2) competencies, the items and the factors suggest a close alignment with the intent of both sets of competencies.

We believe we met the goals of this assessment. This type of assessment is feasible in our setting, as demonstrated by our high response rates. While this is partly explained by the fact that it is mandatory for physicians to participate and that failure to participate can result in loss of license to practice, there is no legislation requiring their respondents (physician or co-worker) to participate. These rates are consistent with the response rates for other groups of Canadian physicians (9–11) who have participated in MSF in conjunction with a regulatory authority.

Almost all of the questions could be answered by the responding physicians and co-workers. We believe that the efforts of the working group who determined the items and the a priori feedback from physicians before the study took place ensured that the items were appropriate for the respondent groups. The items with higher than expected "unable to assess" rates were almost all on the referring physician questionnaire. These data suggest the possible inappropriate selection of respondents or the possible need to delete and/or revise these items that relate to aspects of workplace performance and work and lifestyle balance.

The score profiles are positively skewed. The range and the mean ratings were high, with most physicians receiving all of their ratings between 4 and 5. These profiles are similar to that of other groups (12). Similarly, the self-ratings were lower than those provided by peers, a finding similar to the findings in other studies of this nature (9–11).

Our exploratory factor analyses found that items did group together in factors in ways that are consistent with the intent of the Physician Achievement Review program. The large proportion of variance accounted for in clinical competency on the peer questionnaire reflects the large number of items assessing that component as well as the cohesiveness of the questions (ie, nine questions on reports). Similarly, a large proportion of the variance was accounted for by professional development on the referring physician questionnaire because many questions were asked about that component. Clinical competency was the second factor on this questionnaire (it was first in the radiology colleague questionnaire), accurately reflecting the differences in perspective from these two sources on the radiologists' performance. Moreover, all of the items load heavily on at least one of the factors in each questionnaire, indicating the levels of distinction, cohesion, and theoretical meaning of the identified factors. The College of Physicians and Surgeons of Alberta, as a regulatory authority, is concerned about professionalism, communication, clinical performance, collegiality, and self-management (balance between personal and professional life). As such, the factors we identified provide the general direction for physician improvement, whereas the individual items provide more specific feedback. By providing data by source, the radiologists can assess and compare the information provided by each of the groups. Each physician received descriptive data (means and standard deviations) on the scales and individual items for him- or herself, as well as the group as a whole.

Finally, we have evidence that the instruments are reliable at both an instrument and individual practitioner level. The internal consistency reliability analysis (Cronbach ) suggests that both the instrument and the scales are internally consistent. Furthermore, the generalizability coefficient data indicate that the data provided to each physician were also stable across raters, particularly for ratings provided by co-workers and peers. The instruments had sufficient numbers of items and assessors to provide reliable data. These findings are equivalent to or higher than those found in other studies (7–10).

This study focused on radiologists in one province in Canada. All radiologists in the province were recruited as the program is mandatory. Data for 24 physicians were incomplete when the data collection was closed. However, in work of this nature, the pragmatics of providing feedback to physicians in a timely manner necessitated that comparator data be calculated so the physicians could receive their reports within a reasonable time frame. However, there is little reason to suspect that these physicians had different profiles than the 190 physicians whose data were already available to us. We cannot be certain that radiologists in other parts of Canada or elsewhere in the world would have similar performance profiles. This study, such as all but one MSF study (8), permitted participating physicians to identify respondents. While this may introduce bias, the nature of medical practice makes it difficult for anyone other than the physician to identify those physician colleagues and co-workers who can answer the questions. Furthermore, in a province-wide program involving every physician every 5 years, other approaches to recruitment do not appear feasible at this time. This work is dependent on the honesty of the professionals involved and it is hoped that the assessors have a vested interest in the outcome.

MSF is relatively new. With the advent of the ACGME (1) and CanMEDS (2) competencies, MSF has a certain appeal as a way to inform physicians about professionalism, collegiality, and communication so that they can improve in directed ways. MSF appears to be a relatively inexpensive way of assessing these competencies and assessing the changes physicians make on the basis of the feedback received. While the present study did not examine the use physicians made of their MSF data, this would be a legitimate scholarly inquiry. A follow-up study to determine how the physicians used their data, the changes they made as a result of the feedback, and their perceptions of this type of assessment is certainly warranted and has been undertaken in other MSF work (7,13). At an institutional or provincial level, aggregate data for this group of physicians could be used as part of a needs assessment to guide educational program or institutional policy or procedures. Some discussions have taken place to see how these data can be used to improve care within health regions. While not possible in our setting, correlating MSF data to other more objective data would further establish the evidence for the validity of this approach to assessment.

We believe our MSF instruments for radiology provide a viable way of comprehensively assessing physicians and providing guided feedback on a number of competencies and behaviors. The instruments were developed with a regulatory authority as a quality improvement program. While the items focus on the needs of a regulatory authority, their breadth and scope may provide a base set of items with which to assess aspects of ACGME (1) and CanMEDS (2) competencies for other practicing physicians and residents.

	ADVANCES IN KNOWLEDGE

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 

• The response rates from all participants demonstrate that multisource feedback (MSF) is a feasible way to assess radiologists.
  
• The instruments, developed by a regulatory authority, may be useful as part of an assessment of Accreditation Council for Graduate Medical Education competencies.
  

	IMPLICATIONS FOR PATIENT CARE

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 

• Peers, referring physicians, and co-workers will provide data about patient care that radiologists are able to use to guide changes in practice to enhance patient care.
  
• MSF is a tool that may be useful in quality assurance and patient safety programs.
  
• The self-assessment will require radiologists to reflect on their practice and identify ways to improve patient care.
  

	ACKNOWLEDGMENTS

 
We offer special thanks to Drs Trevor W. Theman, MD, John E. Swiniarski, MBA, and Bryan D. Ward, MD, of the College of Physicians and Surgeons of Alberta for allowing us to continue to be part of this work. We thank Drs Paul W. Burrowes, MD, Tom C. Spiers, MD, Edward J. Wiebe, MD, Bernice M. Capusten, MD, Raymond M. Lewkonia, MB, ChB, and Corinne H. Dyke, MD, who served on the working group.

	FOOTNOTES

 

Abbreviations: ACGME = Accreditation Council for Graduate Medical Education • CanMEDS = Canadian Medical Education Directions for Specialists • MSF = multisource feedback

Author contributions: Guarantor of integrity of entire study, C.V.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, all authors; statistical analysis, all authors; and manuscript editing, all authors

Authors stated no financial relationship to disclose.

	References

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION ADVANCES IN KNOWLEDGE IMPLICATIONS FOR PATIENT CARE References

 

1. ACGME. Outcome project competencies. http://www.acgme.org/outcome/comp/compHome.asp. Updated July 1, 2005. Accessed August 10, 2007.
2. Royal College of Physicians and Surgeons of Canada: the CanMEDS roles framework. http://rcpsc.medical.org/canmeds/index.php. Updated February 2007. Accessed August 10, 2007.
3. Madewell JE, Hattery RR, Thomas SR, et al. American Board of Radiology: maintenance of certification. Radiology 2005;234(1):17–25.[Free Full Text]
4. Donnelly LF, Strife JL. Establishing a program to promote professionalism and effective communication in radiology. Radiology 2006;238(8):773–779.[Abstract/Free Full Text]
5. Halsted MJ. Radiology peer review as an opportunity to reduce errors and improve patient care. J Am Coll Radiol 2004;1(12):984–987.[CrossRef][Medline]
6. Wood J, Collins J, Burnside ES, et al. Patient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills. Acad Radiol 2004;11(8):931–939.[Medline]
7. Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med 2002;77(suppl 10):S64–S66.[CrossRef][Medline]
8. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269(13):1655–1660.[Abstract/Free Full Text]
9. Violato C, Lockyer J. Self and peer assessment of pediatricians, psychiatrists and medicine specialists: implications for self-directed learning. Adv Health Sci Educ Theory Pract 2006;11(3):235–244.[CrossRef][Medline]
10. Violato C, Lockyer J, Fidler H. The assessment of pediatricians by a regulatory authority. Pediatrics 2006;117(3):796–802.[Abstract/Free Full Text]
11. Lockyer JM, Violato C, Fidler H. The assessment of emergency physicians by a regulatory authority. Acad Emerg Med 2006;13(12):1296–1303.[CrossRef][Medline]
12. Archer J, Norcini J, Southgate L, Heard S, Davies H. mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? Adv Health Sci Educ Theory Pract doi:10.1007/S10459-006-9003-3. Published online October 12, 2006. Accessed August 10, 2007.
13. Fidler H, Lockyer J, Toews J, Violato C. Changing physicians' practices: the effect of individual feedback. Acad Med 1999;74(6):702–714.[Medline]
14. College of Physicians and Surgeons of Alberta Physician Achievement Review Program. http://www.par-program.org/PAR-Inst.htm. Accessed August 10, 2007.

This article has been cited by other articles:

				S. Mahgerefteh, J. B. Kruskal, C. S. Yam, A. Blachar, and J. Sosna Peer Review in Diagnostic Radiology: Current State and a Vision for the Future RadioGraphics, June 29, 2009; (2009) 295095086. [Abstract] [Full Text]

This Article Abstract Full Text (PDF) All Versions of this Article: 2473071431v1 247/3/771    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Lockyer, J. M. Articles by Fidler, H. M. Search for Related Content PubMed PubMed Citation Articles by Lockyer, J. M. Articles by Fidler, H. M.