종단적 FD의 효과 평가: 학술성과 연구(Acad Med, 2016)

종단적 FD의 효과 평가: 학술성과 연구(Acad Med, 2016)

Measuring the Impact of Longitudinal Faculty Development: A Study of Academic Achievement

Lori R. Newman, MEd, Stephen R. Pelletier, PhD, and Beth A. Lown, MD

다수의 논문이 기존 의학 교육 펠로우 십의 교과 과정 내용과 과정 전략을 기술하고있다. 그러나 엄격한 프로그램 평가 연구, 특히 결과 데이터를 결정하기 위해 질적 및 양적 방법을 체계적으로 사용하는 조사가 부족합니다 .1,2,15 또한 교육 펠로우쉽 프로그램의 평가는 단일한 데이터, 즉 시간 경과에 따른 변화 또는 비교 집단의 사용에 대한 평가가 부족한 참가자 만족도 조사에서 수집 한 데이터만을 사용하는 서술 방식에 주로 의존했습니다 1,2,4,13,15,16

A number of articles describe existing medical education fellowships’ curriculum content and process strategies3–14; however, there is a paucity of rigorous program evaluation studies, especially investigations that employ systematic use of qualitative and quantitative methods to determine outcome data.1,2,15 Moreover, evaluation of educational fellowship programs has relied predominantly on descriptive methods using single-site data collected from participants’ satisfaction surveys that lack assessment of change over time or use of comparison groups.1,2,4,13,15,16

1998 년 하버드 의과 대학 (HMS)의 교수진은 미국에서 최초의 종적 의학교육 펠로우십 중 하나를 설립했습니다. 이 프로그램의 목적은 다음과 같습니다. 

• 교수로서의 교수 기술을 개발하고 향상. 

• 의학 교육에 대한 학술 연구의 기회 제공; 

• 지도자로서의 동료 지원 및 조직 내의 변화의 대리인; 

• 서로를 지원하고 시스템 전체에 헌신적 인 교육자 커뮤니티를 구축.6,17 

In 1998, members of the Harvard Medical School (HMS) faculty established one of the first longitudinal medical education fellowships in the United States. The program was created to 

• develop and further enhance faculty’s skills as educators; 

• provide opportunities for scholarly research in medical education; 

• support fellows as leaders and agents of change within their organizations; and 

• build a community of educators dedicated to supporting each other and for the system at large.6,17 

펠로우십의 효과성을 평가하기 위해 우리는 프로그램의 개인적 및 전문적 영향을 결정하기위한 2 단계, 혼합 방법 연구를 설계했습니다. 학술 경력의 측정 가능한 결과를 검토하고 시간 경과에 따라 이러한 결과가 어떻게 변하는 지 분석합니다.

To evaluate theeffectiveness of the fellowship, we designeda two-phase, mixed-methods study to determine the personal and professional impact of the program on our fellows and to examine measureable outcomes of their academic careers and analyze how these outcomes change over time. 

이전 연구에서 설명한 바와 같이, 우리는 프로그램 졸업생에 대한 반 구조적 인터뷰를 실시하고 정성 분석을 통해 펠로우쉽의 "필수 요소"가 참가자로서의 개인적 정체성의 변화를 촉진하고 자신감과 자기효능감을 강화 시켰음을 보여주었습니다 .

As described in a previous article,18 in the first phase of the study we conducted semistructured interviews of program graduates and, through qualitative analysis, demonstrated that “essential elements” of the fellowship fostered transformation of participants’ personal identity as educators and enhanced their confidence and self-efficacy.

이 보고서에서는 혼합 방식 연구의 양적 단계에 대해 설명합니다. 10 년 동안의 데이터를 사용하여, 우리는 우리 교육 휄로우 십의 10 가지 측정 가능한 결과를 조사하기 위해 사전 및 사후 이력 (CV) 분석을 실시했습니다.

In this report we describe the quantitative phase of our mixed-methods study. Using 10 years of data, we conducted pre and post curriculum vitae (CV) analyses to examine 10 measurable outcomes of our education fellowships.

Method

연구참가자와 모집

Study participants and recruitment process

The 44 faculty who participated in the fellowship between academic years (AYs) 1999 and 2005 at two HMS sites using a shared curriculum were invited to participate in this two-part study.

졸업생 모집

Recruitment of fellowship graduates.

We sent to the 44 fellowship graduates a letter of invitation, study description, and requests for written informed consent and copies of their updated CVs formatted according to HMS guidelines.

대조군 모집

Recruitment of comparison group.

For the second part of the study, in June 2005 we asked each of the 12 AY2004–2005 HMS fellowship graduates(6 at each of the 2 sites) to nominate one or two physicians to serve as a comparison group. We sent written instructions to the graduates to identify clinician–educators at their institutions who had demonstrated strong interest in clinical teaching. “Interest” was defined as consistent commitment of time and effort in directing or participating in ongoing teaching activities. In addition, we asked the graduates to nominate faculty who were closely matched in terms of the following criteria: gender, number of years since medical school graduation, academic appointment, and clinical discipline. We included specific instructions not to select peers who were graduates of the HMS fellowship program. 

The AY2005 fellows submitted names of 19 nominees, to whom we sent invitations along with informed consent forms and requests for their CVs. Fifteen faculty consented and submitted their HMS-formatted CVs. We reviewed each CV to verify strong interest in medical education and commitment to being active teachers, as well as to determine how closely they matched the fellows according to the criteria outlined above. Twelve faculty peers were chosen as the comparison group. We computed a Mann–Whitney U test to determine whether there were significant differences between the fellows and their peers. All study participants are described in Table 1.

자료 수집, 분석, 접근

Data collection, analysis, and approach

We created a coding manual based on the required HMS CV format to conduct analysis on the collected data (Appendix 1). We chose CV analysis because this represents verifiable data that are detailed and comprehensive.20–23 The HMS CV format requires standard organization and reporting of criteria required for academic advancement. These criteria were derived from a consensus-driven process performed by a faculty task force appointed by the HMS Dean of the Faculty of Medicine. Therefore, academic metrics for this study were the number of longitudinal teaching activities, education- related committee work, teaching presentations, educational leadership roles, medical education funding sources, total publications, medical education publications, teaching awards, newly developed curricula, and academic promotion.

펠로우십 졸업자 자료 수집

Fellowship graduate data collection.

For the first part of the study, we conducted a pre and post analysis of the 42 graduates’ CVs to determine within-subject academic achievement before and after participating in the HMS Fellowship in Medical Education. The starting point for all graduates’ CV analysis was 2 years prior to entering the fellowship. Beginning with the class of AY1999, we collected and entered data at 2-year intervals through June 2009. This provided up to 10 years of data on the earliest fellows.

We compared changes in pre and post outcomes for the fellowship cohort by performing repeated-measures ANOVA. Bonferroni corrections were made to all P values as a conservative measure. Data were entered for all variables two years before fellowship enrollment (F−2), at the end of the fellowship year (F), and at two and four years post fellowship (F+2, F+4).

To provide a conservative estimate of the fellowship’s impact, we removed outlier data from the analysis of publications of one fellow with a large number of publications (163 total publications and 24 medical education publications). We performed further analysis of the 23 participants who were six years post fellowship (F+6) in 2009. We also conducted a cross-tabulation t test analysis to determine the association of academic outcomes with gender and year of medical school graduation.

비교군 자료 수집 

Comparison group data collection.

Using our coding manual, we conducted a pre and post comparative CV analysis of the same academic metrics for the 12 participants in the AY2005 fellowship cohort and the 12 nonfellowship physicians who served as the comparison group. The starting point for the analysis was two years before study entry in 2005; data collection and analysis continued for both groups through June 2009. We computed cumulative outcomes for the fellows and their matched peers to compare academic outcomes. We tested the difference in outcome changes between HMS fellows and matched peers using the Mann-Whitney U test.

결과

Results

Pre and post changes in academic achievement among fellowship graduates

Differences in changes of academic achievement between fellows and comparison group

Discussion

CV 분석은 교수진 개발 프로그램을 평가하기 위해 연구자가 사용한 데이터의 검증 가능한 소스입니다 .1,2,4,15,16,20-23이 방법을 사용한 연구는 참가자의 강의 프레젠테이션, 출판물 및 학술 홍보의 증가를보고했습니다. , 21) 몇몇 연구자들은 통제 집단이 없기 때문에이 연구 결과의 일반화 가능성을 제한하고는 있지만 교육 리더십 활동을 상당히 증가시킨 것으로 나타났다 .5,8,22,23

CV analysis is a verifiable source of data used by researchers to assess faculty development programs.1,2,4,15,16,20–23 Studies using this method have reported increases in participants’ teaching presentations, publications, and academic promotions.20,21 Several investigators have also found significantly increased education leadership activities, although the lack of control groups limits the generalizability of this finding.5,8,22,23

우리는 교육 리더십 또는 교육위원회에서의 academic achievement의 비율과 academic promotion의 긍정적 변화경향성을 발견했다. HMS는 데이터 입력을 끝내기 1 년 전에 교육 및 교육 우수성을 포함하도록 학술 승진 기준을 변경했습니다. 또한 종단적 교육활동, 리더십 역할 지도, 교과 과정 제공, 의학교육위원회, 발표 프레젠테이션 및 의학 교육 출판물 수 등 여섯 가지 학업 측정 지표에서 비교 대상 그룹과 비교되는 그룹을 크게 앞섰습니다.

We found significant changes in fellowship graduates’ rate of academic achievement for teaching leadership roles and medical education committees, and a positive trend in academic promotions. Of note, HMS changed its criteria for academic promotion to include teaching and education excellence only one year before we concluded data entry. Further, fellows significantly outpaced a comparison group of matched peers in six of the academic metrics: longitudinal teaching activities, teaching leadership roles, curricular offerings, medical education committees, teaching presentations, and number of medical education publications.

펠로우십 후 4 년 동안 남성 졸업자보다 여성 졸업생의 의료 교육 투자가 많고 승진률이 높다는 사실은 FD가 역사적으로 지속되온 academic gender disparities를 줄이는데 중요한 전략이 될 수 있음을 시사합니다 .24,25

The finding that female graduates had significantly more medical education funding and higher rates of promotion than male graduates four years post fellowship suggests that longitudinal faculty development may be an important strategy for decreasing historical academic gender disparities.24,25

커크 패 트릭 (Kirkpatrick)의 평가 틀에 따르면, 우리의 연구 결과는 의학 교육에서 1 년 간의 펠로우쉽에 대한 투자가 참가자의 행동에 긍정적 인 변화를 가져오고 제도적 실천을 향상 시킨다는 것을 나타냅니다. 환자, 가족 및 건강 관리 공동체에 대한 프로그램 적 영향을 평가할 효과적인 방법을 결정하기 위해서는 추가 연구가 필요합니다.

In accordance with Kirkpatrick’s evaluation framework, our findings indicate that investment in a yearlong fellowship in medical education produces positive change in participants’ behaviors and improves institutional practice. Further research needs to be done to determine effective methods to assess programmatic impact on patients, families, and health care communities.

22 Moses AS, Skinner DH, Hicks E, O’Sullivan PS. Developing an educator network: The effect of a teaching scholars program in the health professions on networking and productivity. Teach Learn Med. 2009;21:175– 179.

Appendix 1

CV Analysis Coding Manual Developed for the Harvard Medical School Fellowship in Medical Education, From a Study of Academic Achievement and Longitudinal Faculty Development, 1999–2005, Boston, Massachusetts 

All activities are coded by academic year (July 1–June 30, 200X)

Demographic data

• Fellow ID

• Year of medical education fellowship graduation

• Year relative to the fellowship (e.g., F, F+1, F+2)

• Gender

• Year of medical school graduation

• Clinical department

• Appointment at hospital/affiliated institutions

Note: Kirkpatrick level 1—Learner reactions; level 2a—Modifications of attitudes, perceptions; and level 2b—Acquisition of knowledge, skills were reported in a previous qualitative study.18

Kirkpatrick level 3 outcomes: Behavior change

• Number of episodic (“single”) teaching presentations

• Total number of publications (all inclusive)

• Total number of publications related to teaching, medical education, and medical education research

• Funding related to medical education innovation or medical education research

(Note: Count each activity for the first year of funding only)

Kirkpatrick level 4a outcomes: Impact on organizational practice

• Academic appointment

• Number of academic and/or teaching leadership roles including director of a medical school course, clerkship, graduate medical education residency program, clinical division, fellowship curriculum, faculty development program, national continuing medical education course, medical education center, or medical education academy

• Number of major new educational curriculum offerings/materials developed

(Note: Count for the first year of curriculum implementation only)

• Total number of major local, regional, and national committee assignments related to medical education or medical education research

Kirkpatrick level 4b outcomes: Benefits to students, residents, fellows, and faculty

• Number of teaching awards

• Number of longitudinal medical student, resident, fellow, or faculty development teaching activities per year relative to the fellowship

(Note: This includes commitment to teaching in a course or a regularly occurring elective, inpatient attending duties, or other activities that entail consistent 20+ face-to-face contact hours. “Mentoring” hours are not counted because interpretation of this activity varies widely among fellows.)

Kirkpatrick level 4c outcomes: Benefits to patients, communities (not measured)

---

Acad Med. 2015 Nov 24. [Epub ahead of print]

Measuring the Impact of Longitudinal Faculty Development: A Study of Academic Achievement.

Newman LR1, Pelletier SR, Lown BA.

Author information

• 1L.R. Newman is director of faculty education, Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center, and principal associate in medicine, Department of Medicine, Harvard Medical School, Boston, Massachusetts. S.R. Pelletier is senior project manager, Center for Evaluation at Harvard Medical School, Boston, Massachusetts. B.A. Lown is director of facultydevelopment, Mount Auburn Hospital, medical director, Schwartz Center for Compassionate Healthcare, and associate professor of medicine, Harvard Medical School, Boston, Massachusetts.

Abstract

PURPOSE:

Although faculty development programs in medical education have increased over the past two decades, there is a lack of rigorous program evaluation. The aim of this study was to determine quantifiable outcomes of Harvard Medical School's (HMS's) Fellowship in Medical Education and evaluate attainment of its goals.

METHOD:

In 2005 and 2009 the authors collected curricula vitae (CVs) and conducted within-subject analysis of 42 fellowship graduates and also conducted comparison analysis between 12 academic year 2005 fellows and 12 faculty who did not participate in the program. The authors identified 10 metrics of academic advancement. CV analysis for the 42 graduates started 2 years prior to fellowship enrollment and continued for 2-year intervals until June 2009 (10 years of data collection). CV analysis for the comparison group was from 2003 to 2009. The authors also analyzed association between gender and academic outcomes.

RESULTS:

Fellowship graduates demonstrated significant changes in 4 of 10 academic metrics by the end of the fellowship year: academic promotion, educational leadership, education committees, and education funding. Two metrics-educational leadership and committees-showed increased outcomes two years post fellowship, with a positive trend for promotions. Fellowship graduates significantly outpaced the comparison group in 6 of 10 metrics. Women did significantly more committee work, secured more education funding, and were promoted more often than men.

CONCLUSIONS:

Findings indicate that the HMS Fellowship in Medical Education meets programmatic goals and produces positive, measurable academic outcomes. Standardized evaluation metrics of longitudinal faculty development programs would aid cross-institutional comparisons.

PMID:

 

26606720

 

DOI:

 

10.1097/ACM.0000000000001016

[PubMed - as supplied by publisher]