CV분석을 통한 교수개발 성과의 평가(Fam Med. 2000)

CV분석을 통한 교수개발 성과의 평가(Fam Med. 2000)

Evaluating Faculty Development Outcomes by Using Curriculum Vitae Analysis

Jeffrey A. Morzinski, PhD; David B. Schubot, PhD

출판된 FD평가연구는 출판된 연구가 부족하다는 사실 외에도, 방법론적으로 참여자의 만족도와 자기-평가 학습수준만 측정했다는 것을 한계로 지적받는다. 아래와 같은 리뷰가 있었음.

Criticism of published FD evaluation research has targeted shortages of published studies, as well as meth- odological problems with studies that are often limited to measures of participants’ satisfaction and self- assessed learning.

• Reviews by Irby and Hekelman,5

• Sheets and Henry,6

• Meurer and Morzinski,7 and

• Reid et al8

고등교육평가와 연구, 커크패트릭 모델에서 지속적으로 요구된 것처럼, FD프로그램의 성과를 측정하기 위한 방법을 찾고자 했다. Bland 등은 참여자의 CV를 성과 평가로 사용할 것을 제안했다. 그러나 어떻게 체계적으로 CV를 평가할지에 대한 구체적인 방법은 없었다.

Consistent with calls for higher-level evaluation stud- ies and Kirkpatrick’s model,9 we looked for a method to assess FD program outcomes. Bland et al4 propose the use of program participants’ curricula vitae (CVs) for outcome evaluation. However, no methods specify how to systematically assess CVs

CV는 FD프로그램과 FD프로그램 이해관계자들이 중시하는 여러가지 활동에 관한 정보를 담고 있다. 따라서 CV평가는 FD프로그램 및 각 과의 이해관계와 잘 맞는다.

CVs hold information about many of the activities valued by FD programs and their stakeholders. Therefore, CV assessment is compatible with both FD program and departmental interests.

CV를 평가를 위한 기록물로서 사용하는 것은 추가적인 장점이 있다. CV를 분석하는 것이 비침습적 기록-검토 방법이기 때문이므로, 참여자들이 평가도구에 대해서 반응react할 필요가 없다. 프로그램 참여자들은 종종 감당못할 정도로 많은 평가 설문지를 요구받곤 하며, 이는 응답의 편항, 설문시행자에게 좌절, 시간-소모 등의 한계가 있다. 기록물(CV)분석과 같은 비-반응적 평가방법은 응답 편향을 줄여주는 장점이 있으며, 동시에 참가자들이 별도의 평가도구를 완수해야 할 필요성이 없어진다.

Using CVs as evaluation records has an additional advantage. Because CV analysis is a noninvasive record-review method, subjects are not required to re- act to an instrument. Program participants are often inundated with evaluation questionnaires that can be fraught with response bias, frustrating to administer, and time-consuming to complete. Nonreactive evalua- tion methods such as record (ie, CV) review have the benefit of reducing response biases10,11 while eliminat- ing the need for participants to complete another in- strument.

방법

Methods

대상과 프로그램 맥락

Subjects and Program Context

The study subjects were all 17 participants enrolled in the first two cohorts of the Medical College of Wisconsin’s comprehensive 2-year FD program dur- ing the years 1991–1993 and 1992–1994.

총 기간

The overall program length is 2 years.

 

프로그램 기간 중 참가자가 해야 하는 것들

• During each month, participants at- tend a day (about 5 hours) of group workshops and seminars focused on one or more domain areas.

• Be-tween group meetings, participants meet with mentors and program staff who assist them with career advance-ment issues and work on academic projects.

• To suc-cessfully complete the program, participants are re-quired to complete an organizational assessment project and implement or submit to a scholarly forum both an educational and a research project report. 

CV평가를 위한 수집

A standard procedure of our FD program evaluation is to request updated CVs from faculty members 2 years after completion of their FD program. These post-pro- gram CVs were the records used in this study.

 

CV는 세 가지 시기로 나눠서 평가: 프로그램 전 2년, 프로그램 기간(2년), 프로그램 후 2년

To determine the effect of the FD program on par- ticipants, we assigned participants’ CV entries to three time frames.

• The first was the pre-program time frame, which consisted of 2 years (the calendar year prior to participants’ program entry and the calendar year of entry).

• The second was the mid-program time frame (the full calendar year in the middle of the 2 academic years).

• The third was the post-program time frame, which consisted of 2 years (the calendar year of par- ticipant graduation, as well as the following full calen- dar year).

도구 개발

Instrument Development

Based on a content analysis model described by Crabtree and Miller,12 we designed an instrument to code and inventory CV entries. We first developed a template for analyzing CVs, based on the medical school’s recommended CV format. Two program staff jointly analyzed 5 of the 17 CVs using the initial tem- plate and revised the template as necessary to accom- modate entries from those 5 CVs. The initial template design and pilot results were reviewed with the FD project director to ensure congruence between coding categories and program objectives. This use of a priori and iterative category development followed the method described by Constas.13 The category develop- ment process resulted in two products, the first of which was a list of 10 CV codes that would be applied to the remaining 12 graduates’ CVs (Table 1).

The second product from the category development process was a code book that provided inclusion, ex- clusion, and clarifying criteria for coding decisions. During the authors’ process of codebook development, a total of eight criteria were established to guide cod- ing decisions that pertained to the 10 codes of interest in this pilot study. These criteria are listed in Table 2.

 

절차

Procedures

After creation of the template and the code book, the remaining 12 CVs were randomly assigned to the two coders, who followed a systematic procedure for cod-ing them. This procedure included identifying with an “X” any CV entry for which a coding decision was uncertain due to missing or unclear information.

After the coding procedure was followed for all re- maining CVs, semi-structured interviews were con- ducted between the coders and the CV owners. These interviews rely on data review and input by study sub- jects after initial data collection to enhance the accu- racy and thoroughness of qualitative data.10 Interviews followed a written protocol that consisted of

• a descrip- tion of the FD purpose for the study,

• discussion and resolution of all items coded X, and

• opportunities for subjects to ask questions about their own CVs.

 

We agreed that member-check interviews would not be used to solicit unreported products. However, if subjects of- fered new data ), the product would be entered on their CV and coded if it met inclusion criteria.

A final part of the data analysis consisted of a check for inter-rater agreement to determine the degree of concordance between the two CV coders.

Results

평가자간 신뢰도

Inter-rater Reliability

 

총점

Product Totals

 

Discussion

평가 방법에서 기대하지 않았던 두 가지가 있음

At least two parts of this evaluation method’s appli- cation were unexpected.

 

첫째, 주니어 교수들의 CV는 기대보다 조직화와 완결성에 있어서 (개인별로) 차이가 컸다. CV는 우리 기관의 표준CV포멧을 따를 것으로 기대했으며, FD워크숍에서 제공한 가이드를 따를 것을 기대했으나, 대부분은 표준화되지 못한 방법으로 작성되었고, 종종 전반적인 퀄리티를 향상시키기보다는 낮추는 식으로 정리되어 있었다. 또한 불완전하거나 불분명한 entries (CV당 약 3개)가 있었다. 또한 참여자들의 코멘트와 인터뷰 동안의 "product recall"은 일부 활동은 CV에서 빠져있을 수도 있다는 것을 시사했으며, 특히 peer-reviewed presentation이나 위원회 위원 활동 같은 것들이 누락되어있을 가능성이 있었다. 우리의 경섬으로 보건대 CV는 전문가 조언에 따라 향상되지 않으며, 이것이 평가에 활용하는데 제약이 된다.

First, we found greater-than- expected variation in the organization and complete- ness of the CVs prepared by our junior faculty. CVs were expected to be based on our institution’s standard CV format, and guidance on CV preparation was pro- vided in an FD workshop, but most CVs were orga- nized in nonstandard ways, often distracting from, rather than enhancing, overall quality. We were surprised to find incomplete or unclear entries (about three per CV, coded with X). Also, subjects’ comments and “product recall” during interviews suggested that some activi- ties may have been underreported, especially in the ar- eas of peer-reviewed presentations and committee lead- ership activities. Our experience suggests that CVs are not fully improved by expert advice, and their short- comings limit their “as-is” use in evaluation.

두 번째로 놀라웠던 부분은 CV소지자와의 인터뷰에 필요한 시간을 과소평가한 것이다. 인터뷰의 시간은 교수들이 CV를 조직화하는 방법이라든가 어떤 product가 포함되어야 할지 말지를 물어보는 과정에서 더 길어졌다. 우리는 인터뷰 초반에 교수들의 CV development에 대해 컨설팅하는 것은 (비록 평가의 scope은 벗어나더라도) 시간을 좋은 방식으로 사용하는 것이라고 생각했다. 우리가 member-check interview를 미보고된 product를 적극적으로 알아내기 위해서 활용하지는 않았으나, 인터뷰를 통해서 다수의 product를 알게 되었다. 그러나 인터뷰를 자료 수집에 활용하는 것은 기록-검토 방법record-review method의 비-침습성을 훼손하는 것이 된다.

The second surprise in this study was our underesti- mate of the time needed to interview CV owners. The length of the interview was often increased due to faculty member questions about CV organization and whether or not some products should be included. We agreed early in the interview process that faculty con- sultations on CV development, while outside the scope of the evaluation, were a good use of our time. While we did not use member-check interviews to actively solicit unreported products, such interviews may yield greater numbers of products. However, reliance on in- terviews for data collection further undercuts the noninvasive benefit of the record-review method.

연구의 한계

Other limitations are associated with this evaluation method.

• CV data are subject to the limitations of self- report data.

• Second, there are historical threats to this method’s validity. For example, participation in FD activities may have stimulated faculty to revise their strategies for CV entries over the years of interest.

• Third, we did not include a control group, Further, the productivity of our subjects may not have been related to their participation in our FD program, per se.

• A fourth limit of this method is the lag time be- tween the effort to produce a product and the product’s final entry on the CV.

 

---

Fam Med. 2000 Mar;32(3):185-9.

Evaluating faculty development outcomes by using curriculum vitae analysis.

Morzinski JA1, Schubot DB.

Author information

• 1Department of Family and Community Medicine, Medical College of Wisconsin, USA. jmorzins@mcw.edu

Abstract

BACKGROUND AND OBJECTIVES:

Many faculty development (FD) programs depend on external private and public sponsors that routinely require systematic studies on FD outcomes. The results of evaluation studies can influence whether or not programs continue to be funded. To better evaluate program outcomes on academic productivity, this paper presents and illustrates an evaluation method that uses the curricula vitae (CVs) of FD program graduates.

METHODS:

The evaluation method is implemented by first preparing a record-review template of coding categories that is applied to FD graduates' updated CVs. Next, semi-structured interviews are held with subjects to resolve uncertain CV codes. Finally, coded data are entered into standardized forms and analyzed to yield descriptive findings. The method was piloted with two groups of FD graduates (n = 17) to determine its utility and limitations.

RESULTS:

Results show excellent inter-rater reliability (Cohen's Kappa = .79). There was an overall increase in productivity, measured by the CV, during and after the FD program.

CONCLUSIONS:

CV analysis can be a useful method for assessing FD program outcomes. Several limitations of the method, such as incomplete CVs and self-report bias, must be considered.

PMID:

 

10726219

 

[PubMed - indexed for MEDLINE]