임파워먼트 평가 : 의과대학 교육과정 변화와 평가를 위한 협력적 접근법

임파워먼트 평가 : 의과대학 교육과정 변화와 평가를 위한 협력적 접근법

Empowerment Evaluation: A Collaborative Approach to Evaluating and Transforming a Medical School Curriculum

David M. Fetterman, PhD, Jennifer Deitz, MA, and Neil Gesundheit, MD, MPH

의과대학 교육과정 변화의 어려움

• Changing a medical school's curriculum is commonly an arduous process because faculty, students, and administrators may hold divergent opinions about both the goals for medical education and the best ways to achieve those goals.1,2 
• In addition, curriculum reformers may face stiff resistance if they challenge deeply entrenched interests, traditions, and institutional culture.3,4 
• Proposals for change may engender debate that can be rancorous and adversarial among competing faculty, such as basic scientists and clinicians.5 

그러나 철학적, 현실적 합의에 도달하는 것은 중요함.

Yet, reaching a philosophical and practical consensus for change is critical, because a unified vision is needed to achieve broad acceptance, coordinated implementation, and subsequent refinement of a new curriculum.6 

저자들의 두 가지 질문

(1) 교육과정의 이상적 내용에 대한 합의에 도달하는 가장 좋은 방법은?

(2) 지속가능한 평가 주기와 향상을 위한 교육과정 평가의 최선의 방법은?

We (i.e., the authors, who formed the evaluation team) ask two important questions: (1) What are the best methods for achieving consensus about the optimal content of a medical curriculum? and (2) What is the best way to evaluate a curriculum to ensure that it will undergo sustainable cycles of review and improvement?

지난 20년간, 기초과학 지식의 향상과 의학교육의 혁신에 의하여 많은 미국 의과대학 교육과정은 크게 변화하였음. 성공적으로 교육과정 변화를 이뤄낸 학교들에 대한 연구로부터 Bland 등은6개의 핵심 특징을 찾아내였음. 또한 Loeser등은 다섯 개의 교육과정 변화의 단계를 주장하였고 그 마지막 단계는 도입과 평가이다. 대부분의 교육과정은 그것이 처음 도입된 시점에서부터 완벽한 경우는 거의 없기 때문에 효과적인 평가는 교육과정 개혁의 핵심이라고 할 수 있다. 실제로 새로운 교육과정을 설계한 사람들조차 그들이 생각하는 이상적인 모습에 도달하기 위하여 비판과 교정의 과정을 바라고 있다.

During the past 20 years, stimulated by the growth of basic science knowledge and by innovations in medical education, most U.S. medical schools have implemented significant changes in their undergraduate curricula.7 In a review of schools that had undergone successful curricular change, Bland and colleagues 8 identified six predominant features: 

• leadership, 
• a cooperative climate, 
• participation by organization members, 
• favorable politics, 
• human resource development, and 
• effective evaluation. 

Similarly, Loeser and colleagues 6 described five phases of curriculum change that were used at the University of California, San Francisco; the last phase was the implementation and evaluation of the new curriculum. Thus, effective evaluation is recognized as a key component of curricular reform, in part because most curricula are far from perfect when first implemented. Indeed, those who devise a new curriculum depend on and anticipate cycles of critique and revision to approximate the ideal.9–12

스탠포드 의과대학의 교육과정 변화

The Stanford University School of Medicine underwent a major change in its undergraduate medical education when a new curriculum was introduced in academic year 2003–2004. The curriculum continued to be refined yearly from 2004 through 2008. We found the principles of “empowerment evaluation”13–15—an innovative evaluation method that involves stakeholders in an egalitarian process of review, critique, and improvement—to be useful in evaluating the curriculum and in guiding change. The empowerment evaluation method includes a process for ongoing scrutiny of the effectiveness of the curriculum along with the promotion of a cooperative atmosphere. In this paper, we describe the key characteristics of empowerment evaluation and show how using its tools has helped faculty, students, administrators, and evaluators (as a group, referred to here as “stakeholders”) to revise, refine, and improve our medical school curriculum.

Empowerment Evaluation: The Approach and Its Tools

The approach

Empowerment evaluation, broadly defined, is an approach to gathering, analyzing, and using data about a program and its outcomes that actively involves key stakeholders in all aspects of the evaluation process. The approach provides a strategy by which and a structure within which stakeholders can empower themselves to engage in system changes; this process is based on the assumption that the more closely stakeholders are engaged in interpreting, discussing, and reflecting on evaluation findings in a collaborative and collegial setting, the more likely they are to take ownership of the results and to use evaluation to guide curricular decision making and reform.

 

The tools

Five tools are integral to the empowerment evaluation approach.15 We discuss each one below.

 

"근거의 문화" 만들기

Developing a culture of evidence.

Emphasis is placed on rapidly disseminating formative feedback and data about the curriculum to help inform decision making. Reporting processes are established that will improve stakeholder access to evidence (evaluation data and findings) and that will provide regular opportunities for reviewing and discussing the evidence. Faculty are encouraged and supported in their efforts to use evaluation evidence as a tool to support or refute their positions in debates about curriculum. By making this practice a cultural norm within the institution, a culture of evidence is established.

 

"비판적 친구" 활용하기

Using a critical friend.

The empowerment evaluation method uses a professionally trained evaluator to help facilitate evaluation practices at an institution. However, the role the evaluator plays is not that of an outsider or an expert removed from curriculum development and implementation but, rather, that of a trusted colleague—a critical friend—who can coach stakeholders as they engage in self-evaluation, strategic planning, and curriculum implementation. (Critical friend의 역할)

• This critical friend can help set the tone for discussions about evaluation results and findings, by modeling and informally articulating the philosophy behind the empowerment evaluation approach.16–20 
• Specifically, the critical friend should help to establish a positive learning climate in which the views of all stakeholders are respected, input from all parties is solicited, and the conversation is guided so as to encourage comments that are constructive and improvement oriented. 
• Critical friends model a communication style that is inviting, unassuming, nonjudgmental, and supportive, so that stakeholders will feel comfortable in speaking openly about curricular issues and concerns. 
• Critical friends should also help to remind members of the group about what they have in common, including shared curricular goals and institutional values.

 

"성찰과 행동의 사이클 장려하기"

Encouraging a cycle of reflection and action.

Structured opportunities allow stakeholders to reflect on evaluation data, set priorities, and develop action plans for revising the courses and curriculum. Once revisions are adopted and implemented, those too are monitored and evaluated. The cycle of inquiry and evaluation is thus ongoing and continuous.

 

학습자의 커뮤니티를 활성화시키기

Cultivating a community of learners.

Under the empowerment evaluation model, the assumption is that each stakeholder, regardless of his or her status within the institution, has a unique and important perspective on curricular issues that should be shared and valued. As stakeholders engage in dialogue and discussion, they learn from one another and, in the process, are able to broaden and deepen their understanding of issues affecting individual courses and, more broadly, the curriculum. They also gain greater insight into the steps involved or hurdles encountered in making improvements and revisions. Emphasizing learning and discovery in a collaborative and egalitarian environment helps to make discussions about evaluation findings more empowering and less threatening. Reaching a consensus cooperatively also helps to reduce the likelihood that mandates for reform will be imposed on faculty or students in a top-down or hierarchical fashion.

 

자기성찰을 할 수 있도록 하기

Developing reflective practitioners.

Building in structured time, assignments, and formal expectations for all stakeholders helps to immerse stakeholders in the evaluation process. Faculty, students, and administrators develop the habit of continually reflecting on their practices and programs, both individually and as part of a larger group or community. By establishing a habit of regular self-assessment, stakeholders become reflective practitioners.

These tools of empowerment evaluation are mutually reinforcing, and they frequently operate together. However, it may be useful, at first implementation of this approach, for those involved to think of the progression as taking place in stages. The stages build on each other, generating a momentum, which culminates in the development of a more refined and constructive evaluative eye (Figure 1).

 

How Empowerment Evaluation Comes Together

 

The theoretical framework and the tools integral to the empowerment evaluation approach, as described, provide the basis for establishing and routinizing a system of evaluation that can be applied across the medical school curriculum. At Stanford, the stakeholders initially focused on developing this evaluation system at the preclinical level.

1. First, data were assembled from midquarter focus groups, classroom observations, and end-of-quarter student questionnaires. 
2. Second, faculty members received these data in advance of a group meeting; at the meeting, faculty members discussed the positive and negative data from the evaluations, their own reflections, and the proposed next steps. A trained evaluator, who served as a critical friend, mediated the meeting, providing data and support but also being willing to highlight weaknesses in the course(s) and approach. This facilitated process helped to build a culture of evidence, in which it was the norm to use data to support one's position and inform decision making. 
3. Third, the faculty members engaged in an initial cycle of reflection and action by discussing ways in which they could collaborate between courses to enhance student learning—for example, by improving the logical sequencing of topics across the curriculum and by removing unintended redundancies. The evaluation findings and faculty responses served to stimulate discussion at curriculum committee meetings and in the faculty senate, which helped to cultivate a community of learners across the school. 
4. Fourth, faculty and student task forces worked together to complete the design and implementation of the revised courses and/or curriculum. 
5. Fifth, the process came full circle: The courses and curriculum were evaluated by both students and faculty to determine the effectiveness of the new educational interventions and curricular modifications. 
6. This last phase is the full cycle of reflection and action that is most characteristic of empowerment evaluation. Through this process, faculty, students, and administrators enhanced their goal of becoming reflective practitioners, who routinely assess teaching and learning in the school.

Application of Empowerment Evaluation Principles and Tools in the Evaluation of a Medical School Curriculum

 

The challenge of transitioning students and faculty through a major restructuring of the curriculum at Stanford lent itself to a parallel reassessment of the existing evaluation methods. The stakeholders, led by the evaluation team, began to think more explicitly about the values that were implicit in our practices and to explore ways to address key weaknesses in our approach.

 

기존의 접근법 Prior approach to evaluation and the areas needing change

OME의 문제

In the past, the stakeholders and the Office of Medical Education (OME) did not explicitly articulate a set of values or goals to frame our approach to evaluation. Implicitly, we were guided by a tradition within the institution that each course was a discrete entity and that responsibility for that course lay primarily with the individual course director. (...)

이혜해관계자의 참여를 지원할 공식적 구조와 절차의 부족

Other weaknesses in our prior approach were a lack of formal structures and processes to encourage and support stakeholder engagement in the evaluation process. Within OME, we did not have dedicated staff members in charge of developing the evaluation program or managing and reporting evaluation data. Consequently, although the OME staff were vigilant about collecting evaluations from students at the end of each course, in some cases, data were “warehoused,” and evaluation results were not made available to course or clerkship directors for months or even years afterward.(...)

전국적으로 벌어지고 있는 의학교육의 변화를 잘 반영하는 평가법을 도입하고자 함.

"평가의 문화가 새로운 교수-학습 문화와 consonant하기 위함"

In identifying and instituting a new model of evaluation, we wanted an approach that would mirror the changes taking place in medical education nationwide.2,7,21,22 Our new medical school curriculum emphasized an integrated and applied approach to teaching and learning, and, in this new environment, faculty needed to work more collaboratively to develop the curriculum. Developing a single block of lectures in the integrated-organ-systems course on the heart or lung required input from and involvement of faculty in the departments of histology, pathology, microbiology, physiology, and pharmacology. Similarly, the stakeholders now expected more participatory involvement on the part of both teachers and learners as active learning exercises were developed in which preceptors would serve not as content experts but as facilitators, helping teams of students diagnose and develop treatment plans for patient problems.23 It was our hope that the new model of evaluation would reflect these changing institutional values and practices and, thus, would bring collaboration and participation to the forefront. The culture of evaluation would be consonant with the new culture of teaching and learning.

 

현재 접근법 Current approach to evaluation

 

구조화된, 일관성있는 프로그램 도입

The new Division of Evaluation staff worked to implement a structured, coherent program that would apply empowerment evaluation principles and tools in evaluating the effectiveness of the curriculum. 

• Empowerment evaluation tools served to address a wide range of issues affecting the educational program and curriculum at Stanford, including issues surrounding courses, clerkships, scholarly concentrations, advising, and faculty instruction. 
• The approach also helped us to address crosscutting curricular issues such as scheduling, standardization of syllabi, and clinical skills training. 
• Finally, the stakeholders used empowerment evaluation to identify broader programmatic issues, such as the need for changes in staffing and the resource allocation needs for integrated courses.

학생과 교수들이 교육과정에 대한 지속적인 대화에 정기적으로, 구조화(structured)되어 참여할 수 있는 기회제공.

Initially, however, the primary focus was on evaluating and monitoring the preclinical curriculum, because this aspect of our program was undergoing the most radical transformation. Shifting to an applied, integrated curriculum meant changes in the faculty's roles and responsibilities. Course content and hours were adjusted, and some faculty had great concern that eliminating content in some areas in order to expand content in others might jeopardize students' knowledge base and preparedness as researchers and clinicians. We therefore felt that it was important to establish regular, structured opportunities for students and faculty to engage in an ongoing dialogue about the curriculum, to reflect on strengths and weaknesses, and to collaboratively seek solutions when problems were identified.

 

임파워먼트 평가 모델을 적용하기

Applying the empowerment evaluation model to the preclinical curriculum

 

• We applied several familiar evaluation tools to our review of the preclinical curriculum, including focus groups and surveys. (...) 
• Applying the empowerment model at Stanford also required improvements in the data-collection and -reporting processes. (...) with empowerment evaluation, we distilled and analyzed qualitative and quantitative data, using both standard questions across the curriculum and questions rooted in specific concerns of course directors. (...) By emphasizing the positive steps that could be taken to improve a course, evaluators sought to minimize the disillusionment that can sometimes accompany poor student ratings and to provide encouragement and support for faculty as they worked to revise and refine courses, curriculum, and the quality of instruction. 
• In the past, scheduled meetings with course directors were rare or sporadic. (...). These meetings provided structured opportunities for what Bohm 25 called “dialogic engagement,” or exchanges among course directors, students, the curriculum committee, and the empowerment evaluators. 
• As access to evaluation data improved and the process of sharing evaluation data and results became less threatening, faculty became more comfortable engaging in dialogue about evaluation findings. 
• Working closely with many individual course directors, students, and administrators and adopting an inclusive and pluralistic perspective toward data collection, the evaluation team was in a unique position to identify patterns across the curriculum.(...) Thus, a continuous and systematic cycle of reflection and action across the curriculum became institutionalized.  
• Collaborative efforts at all levels contributed to providing a comprehensive self-assessment of the curriculum and placed stakeholders back in the driver's seat—providing them with forums and opportunities to become leading voices in setting priorities. Through continuous engagement in the evaluation process and daily immersion in evaluation discourse, stakeholders learned to mirror the process in their daily practice, thus becoming reflective practitioners.

 

Measuring the Effectiveness of Empowerment Evaluation

 

Internal metrics

 

Applying the empowerment evaluation model to curriculum evaluation at Stanford has contributed to improvements in course and clerkship ratings. In comparing evaluation results before and after stakeholders began using this approach, we found that the average student ratings for the required courses (18 courses) improved significantly (P = .04; Student's one-sample t test), as shown in Figure 2. Similarly, ratings for most of the required clerkships remained steady or improved. When dialogue around evaluation findings led to the development of targeted strategies for addressing key weaknesses in courses, clerkships, or the overall curriculum, student ratings showed a marked improvement. Three examples that stand out in this regard are the “Cells to Tissues” course, the obstetrics–gynecology and surgery clerkships, and our cross-curricular approach to clinical skills training.

Figure 2

 

External metrics

 

Finally, we explored the use of several external metrics to measure the effectiveness of the new Stanford curriculum as it was being refined by empowerment evaluation. 

• The median United States Medical Licensing Examination (USMLE) Step 1 score at Stanford for the three years before the introduction of the new curriculum and evaluation method was 230, whereas the median score rose to 237 by three years after the introduction. 
• The failure rates of students in USMLE Step 1 and USMLE Step 2 Clinical Knowledge, both before and after the curricular change, were each under 2% (national rates: 6% and 5%, respectively). 
• For clinical skills development, no data were available from 2000–2003 because USMLE Step 2 Clinical Skills was introduced for the first time in 2004–2005. However, in academic years 2005–2006, 2006–2007, and 2007–2008, the failure rate on this exam at Stanford was 1%, whereas the national failure rate was 3%. 
• We also examined student performance during the first postgraduate year of training. Approximately 80% of our graduating students match at competitive academic residencies. In their first year of residency, our graduates' performance generally met or exceeded expectations in skills and knowledge domains, as reported by residency program directors. 
• Although these metrics lack the precision and certainty of a controlled experiment, the aggregate measures indicate that student education was kept at a high level or was enhanced by the introduction of the new curriculum and the simultaneous use of empowerment evaluation to refine it.

Challenges and Barriers

 

Adopting an empowerment evaluation approach at a medical school is rewarding but also challenging. Faculty members...

• often have demanding schedules and little frame of reference for participating in collaborative feedback. 
• Time devoted to reviewing the curriculum and modifying courses may compete with time devoted to research and publication—activities that commonly bring greater institutional rewards. 
• Some faculty, accustomed to reviewing their evaluation results in private, initially had concerns about discussing course and curriculum strengths and weaknesses in a group setting. 
• For empowerment evaluation to work effectively, members of both the administration and the faculty needed to view their participation in teaching and a candid critique of the school's curriculum as priorities equal to research, publications, and grants.

(...)

Empowerment evaluation requires an investment of resources. 

• Institutions must be committed to supporting the evaluation staff with formal training to facilitate the process, and 
• faculty time must be protected to allow their participation in meetings about and in discussions of the curriculum. 
• In the absence of an evaluation unit and a regular meeting schedule, the collaborative process can be compromised. 
• If data are not collected, analyzed, disseminated, discussed, and reviewed in a timely manner, rigor may not be maintained.

토마스 에디슨은, "많은 사람들이 기회를 놓치곤 하는데, 왜냐하면 기회는 작업복을 걸치고 있어 일처럼 보이기 때문이다."

Thomas Edison once observed, “Opportunity is missed by most people, because it is dressed in overalls and looks like work.” (...)

 

Receptivity of Stakeholders to Empowerment Evaluation

 

A department chair said, 

We are grateful [to the empowerment evaluation approach] as we continue to improve our departmental teaching programs! In addition to the superb efforts [of our clerkship directors], our entire faculty and our residents and fellows are now much more committed to and involved with medical student education.

 

The director of clerkship education said,

Several clerkships have responded to this [evaluation] feedback by requesting individualized coaching [from the evaluation group, which has] made several site visits to assist clerkship directors in revising their orientations, syllabi, and student performance evaluations. End-of-clerkship evaluations demonstrate improved ratings as a result of feedback and coaching; narrative performance evaluations have improved in terms of completeness and level of detail.

 

The director of a medical student clinical clerkship said, 

“All of the Stanford sites want to help improve their teaching. The differences [that empowerment evaluation] made at our neighboring hospitals were visible and clearly effective.”

---

Acad Med. 2010 May;85(5):813-20. doi: 10.1097/ACM.0b013e3181d74269.

Empowerment evaluation: a collaborative approach to evaluating and transforming a medical school curriculum.

Fetterman DM1, Deitz J, Gesundheit N.

Author information

Abstract

Medical schools continually evolve their curricula to keep students abreast of advances in basic, translational, and clinical sciences. To provide feedback to educators, critical evaluation of the effectiveness of these curricular changes is necessary. This article describes a method of curriculum evaluation, called "empowerment evaluation," that is new to medical education. It mirrors the increasingly collaborative culture of medical education and offers tools to enhance the faculty's teaching experience and students' learning environments. Empowerment evaluation provides a method for gathering, analyzing, and sharing data about a program and its outcomes and encourages faculty, students, and support personnel to actively participate in system changes. It assumes that the more closely stakeholders are involved in reflecting on evaluation findings, the more likely they are to take ownership of the results and to guide curricular decision making and reform. The steps of empowerment evaluation include collecting evaluation data, designating a "critical friend" to communicate areas of potential improvement, establishing a culture of evidence, encouraging a cycle of reflection and action, cultivating a community of learners, and developing reflective educational practitioners. This article illustrates how stakeholders used the principles of empowerment evaluation to facilitate yearly cycles of improvement at the Stanford University School of Medicine, which implemented a major curriculum reform in 2003-2004. The use of empowerment evaluation concepts and tools fostered greater institutional self-reflection, led to an evidence-based model of decision making, and expanded opportunities for students, faculty, and support staff to work collaboratively to improve and refine the medical school's curriculum.

PMID:

 

20520033

 

[PubMed - indexed for MEDLINE]