졸업후의학교육(GME)에 대한 프로그램 평가의 개념적 모델

졸업후의학교육(GME)에 대한 프로그램 평가의 개념적 모델

A Conceptual Model for Program Evaluation in Graduate Medical Education

David W. Musick, MA, PhD

GME는 새로운 시대에 접어들었다. 

Graduate medical education (GME) has entered into a new era, one that has been described as a “paradigm shift.”1,2 Accreditation standards adopted in recent years by the Accreditation Council for Graduate Medical Education (ACGME) have stressed the importance of program evaluation as part of an overall shift from a process-oriented to an outcomes-oriented system of education. (...) following statement taken from ACGME accreditation standards: “The program should use resident performance and outcome assessment in its evaluation of the educational effectiveness of the residency program.”3

outcome에 대한 강조는 GME가 어떠해야하는가에 대해서 여러가지 시사점이 있다. 그러나 educational effectiveness는 무엇이며, 그것을 어떻게 측정할 것인가?

This new emphasis on outcomes has many implications for how GME training programs function, and perhaps for how the process of program accreditation site reviews will function in the future. But what is meant by “educational effectiveness” and how does one evaluate it?(...)

(...). It is apparent that there is not a shared definition of the term program evaluation, particularly when it is associated with terminology referring to educational outcomes.4

Evaluation이란 단어가 갖는 의미의 혼동. 

Part of the confusion may be an issue of semantics. The term evaluation is often used interchangeably with the term assessment. And evaluation is used broadly within medical education, and can refer to several distinct processes: 

overall evaluation of training programs as a whole; 

of an individual resident’s performance; 

of faculty teaching; 

or of a given educational lecture, conference, rotation or other learning experience within a training program. (...)

본 논문의 목적

The purpose of this article is four-fold:

• * to briefly review the literature pertaining to program evaluation, both in general terms and in reference to medical education,
• * to present a task-oriented conceptual model of program evaluation,
• * to discuss outcomes evaluation as one type of program evaluation (distinguishing between relevant institutional and program standards), and
• * to provide a five-step process that will assist program directors and/or other medical educators in developing effective ways to use program evaluation data to improve GME training programs.

문헌 조사

Review of the Literature

 

Evaluation의 정의와 역사적 흐름

The term evaluation is best defined as a process of decision making about the object being evaluated and how it compares to some standard of acceptability.5 (...). Much of the evaluation literature is found within the realm of the social sciences. Evaluation as a distinct discipline gained impetus in the early 1970s as part of the Great Society movement. (...) An early evaluation model described program evaluation as a process where participants agreed in advance on the purpose and design of evaluation procedures, and on how the results would be used.6

교육적 상황에서 프로그램 평가

Program evaluation in the educational setting began to receive more emphasis during the late 1970s and 1980s as a result of increased governmental funding of reform initiatives at all levels of education. (...) Most often, evaluation in an educational setting was conceptualized as a process of making decisions about whether an educational program was meeting its goals and objectives. (...)

의학교육에서의 프로그램 평가

Regarding program evaluation in medical education, the literature pertaining to work done with medical schools and residency programs is less developed than for other fields of education and is largely descriptive.10 Evaluation models specific to GME programs have been nonexistent; there is simply no overarching theoretical base or consistent approach provided whereby GME program directors can determine what is expected in this regard. (...) 

프로그램 평가의 개념적 모델

Conceptual Model of Program Evaluation

 

List 1 provides a task-oriented conceptual model for evaluation within a GME training program. The central notion of the model is to identify the steps involved in planning and carrying out various types of evaluation, consistent with evaluation best practices and accreditation requirements. 

The evaluation need and focus represent the initial stages of determining why the evaluation is to be done, what or who is to be evaluated and what “rules” or standards will inform the evaluation. 

The evaluation methodology is the stage where procedures are established for how to collect and analyze evaluation data. 

Finally, the evaluation results stage represents the presentation of data to key stakeholders in an established forum (such as an annual program evaluation meeting). and also the written documentation of all steps taken in performing the evaluation as well as decisions made as a result. 

성과평가란 무엇인가?

What is Outcomes Evaluation?

 

ACGME의 정의

Outcomes evaluation refers to a particular type of program evaluation. It is defined by the ACGME as

"evidence showing the degree to which program purposes and objectives are or are not being attained, including achievement of appropriate skills and competencies by students.11"

다른 평가법과의 차이

The primary distinction between an outcomes-oriented approach and other approaches to evaluation is found in the word evidence. In GME, accreditation reviews have traditionally focused on the process of education.(...) This emphasis on process occurred because of the dominance of the Flexnerian model of medical education and because of the difficulty in defining competence in precise terms within a given discipline.12

최근 six general competencies와 함께 outcome에도 관심을 갖게 됨

In recent years, especially with a shift to an educational framework based on the “six general competencies” of GME, accreditation has now begun to emphasize not only the educational process but also its outcomes, (...)

ACGME는 성과평가에 대하여 아래와 같이 강조하고 있음.

The new emphasis on outcomes evaluation is illustrated by the following statement of the ACGME:

"Assessing the actual accomplishments of a program requires a different set of questions: (1). Do the residents achieve the learning objectives set by the program? (2). What evidence can the program provide that it does so? (3). How does the program demonstrate continuous improvement in its educational processes?13"

성과평가를 위한 두 가지의 교육적 요소가 있음

To illustrate this new emphasis on outcomes evaluation, in the following paragraphs, two common educational components of all GME programs will be considered, one simple and the other more complex.

첫 번째: didactic instruction자체의 process와 outcome을 같이 평가함.

As a simple illustration of documenting an educational program outcome, consider that all ACGME-accredited training programs are required to provide didactic (i.e., lecture-based) instruction in an organized fashion. (...)

Based on an outcomes-oriented approach to accreditation, a site visitor will review the training program’s didactic program focusing on two things: 

• the process of education (e.g., What lectures are provided and when? Lecture topics? Comprehensiveness? Organization?); and, 
• the programmatic outcome pertaining to the didactic schedule (e.g., Did these lectures actually occur? Who showed up for them? Who presented them?). 

The point of this simple illustration is this: under an outcomes approach to program evaluation, both types of information (i.e., process and outcome) will likely be necessary to satisfy accreditation requirements pertaining to resident didactics.

두 번째: didactic instruction이 실제로 individual resident와 program performance를 향상시켰는지 평가함

A second, more complex example will also illustrate the difference between process-oriented and outcomes-oriented accreditation procedures. (...) Is this fact sufficient for accreditation purposes? More than likely, it is not. After all, residents could be attending didactics but still not learning! (...) Answering these questions satisfactorily results in what the ACMGE labels “evidence of how educational outcomes data is used to improve individual resident and overall program performance.”13

성과바탕 접근법에 기반하여, ACGME는 개별 레지던트의 역량을 측정하기 위한 기대 수준을 상당히 올려두었다. apprenticeship model으로 가르칠 수는 있겠지만, faculty와 충분한 시간을 보낸다는 것만으로는 개별 전공의의 역량을 보여주는 것으로 충분치 않다. 

(...)With the advent of an outcomes-oriented approach, the ACGME has considerably raised the expectation level regarding teaching and measuring the competency of individual residents. Put simply, each training program must redesign its curriculum around the six general competencies and must put into place educational assessment procedures that will effectively document that residents’ learning has taken place, and that such learning has positively affected patient care. This will require programs to institute measures of residents’ knowledge, skills and attitudes in a more formal way than has previously been done. While the apprenticeship model will continue to be valuable from a teaching process standpoint, the presumption that residents have gained sufficient clinical competence by spending time with attending faculty over the course of the training program is no longer acceptable for the purpose of documenting an individual resident’s competency under an outcomes-oriented model of accreditation.

 

GME의 outcome measure

List 2 provides a list of categories of educational outcome measures that can be used by residency directors to document (i.e., provide evidence of) residents’ learning and/or program success.(...)

이러한 달라진 접근법은 GME 수련 프로그램의 '결과물', 의학교육 연속체의 연속성 증대, 그리고 궁극적으로 더 강력한 의료전문직을 강조하고 있다.

The ACGME has stated that this modified approach to accreditation review, which consists of examining both the process and the outcomes of the GME training program, will result in stronger residency training, increased accountability for the “product” of GME training programs (i.e., the competent physician), greater continuity between various levels of the medical education continuum and, ultimately, a stronger medical profession.14

 

성과 평가는 상당히 맥락의존적이다.

Outcomes evaluation is highly context-dependent, in that the expectations and needs of various constituents involved in the program being evaluated must be considered. (...) A key part of any outcomes evaluation system is to determine what outcomes are to be measured and who is to select those outcomes. In residency programs, measurable outcomes are gradually being added to disciplinary ACGME standards by the respective RRCs. National discipline-specific groups (e.g., specialty certification boards, residency program director associations) may also contribute to discussions about appropriate educational outcomes for GME training programs.

기관별 기준 또는 프로그램별 기준?

Institutional or Program Standards?

 

또한 성과에 대해서 기관 차원의 기준과 프로그램 차원의 기준이 다를 수 있음도 염두에 두어야 한다.

It must also be remembered that there is a difference between institutional and program standards related to evaluation outcomes. (...)

Sponsoring institution은 teaching hospital과 그 병원의 GME프로그램들이 institutional standard에 신경을 쓰도록 해야 한다.

There are certain institutional standards to which the sponsoring institution must ensure sufficient attention by the teaching hospital and all its GME programs. (...) Institutions are expected to work closely with individual programs to develop educational objectives and methods of measuring educational outcomes in each of the six competency categories (i.e., medical knowledge, patient care, practice-based learning/improvement, interpersonal and communication skills, professionalism and systems-based practice). This dual accountability for educational outcomes means that a more centralized approach to managing GME will be needed to achieve compliance with all applicable ACGME standards. (...)

Applying the Conceptual Model of Evaluation

(...)

Conclusion

 

In this article, I have discussed how the outcomes approach to education has influenced program evaluation procedures in GME 

• by reviewing the literature; 
• by offering a conceptual model of evaluation that emphasizes systematic, rigorous attention to evaluation need, focus, methods, and results; 
• by distinguishing between programmatic and institutional outcomes; and 
• by outlining a stepwise process of evaluation design and implementation. (...)

이런 새로운 이론적 접근을 활용하여 GME의 독특한 세팅에 어떻게 적용할 것인가에 대한 전문가가 필요하다.

With this new theoretical approach has come a need for additional expertise in how evaluation procedures are applied to the unique setting of GME. (...).

성과 중심으로 옮겨가는 이러한 과정에 비판이 없는 것은 아니다. 

The outcomes movement in education is not without its critics. (...) There is philosophical tension between the need to extensively document educational outcomes via the methods I have described here, and the need to encourage resident physicians to be self-motivated, independent learners. The previous example of documenting lecture attendance is germane here. How can we encourage self-motivated, self-directed adult learning among resident physicians while we take attendance at every lecture as if our learners are still in grade school? (...)

하지만 수년간은 이러한 접근법이 유지될 것으로 보인다. 이미 많은 부분이 성과중심으로 진행되고 있기 때문이다. 

It appears likely that the evidence-oriented approach represented by the educational outcomes movement will remain predominant for years to come. 

• Indeed, the outcomes approach (with particular emphasis on the use of the six general competencies) has also been adopted as a substantial part of continuing medical education 17 and physician certification and recertification 14 processes. 
• It is also prominent within the accreditation process for medical schools, as evidenced by language contained in standards promulgated by the Liaison Committee for Medical Education: “Educational objectives state what students are expected to learn, not what is to be taught … student achievement of these objectives must be documented by specific and measurable outcomes.”18

종합 정리, 추가적으로 필요한 것.

One GME leader has stated that “the ACGME is interested in the competency of the training program and whether the program has demonstrated a pattern of graduating individuals who are competent.”14 This is indeed an important goal and one that deserves support. However, achieving this goal will require two additional strategies. 

• One, institutions that sponsor GME must recognize that the outcomes framework (whereby training programs are expected to formally measure and document individual resident competency) represents a major educational paradigm shift within residency training. Achieving the goal of increased competency of graduating resident physicians will require additional resources devoted to faculty development, curriculum planning and competency measurement. 
• Two, as the ACGME continues to develop its final procedures for determining whether training programs are adequately measuring competency in its graduates, it must respond to criticism by many GME program directors that the expectations in this regard are open to subjective interpretation. As a DIO, I frequently saw instances where programs in certain disciplines were held to higher documentation standards than programs in other disciplines; or where various programs within a given discipline seemed to receive different accreditation results in spite of similar approaches taken to program evaluation issues. 

전공과목별로 어떤 성과를 측정할 것인가에 대한 Flexibility는 바람직하나, arbitrariness는 부적절하다.

Flexibility among various specialty disciplines in choosing which outcomes to measure (and methods used to measure them) is desirable; but arbitrariness in accreditation decision making based on a lack of consistent understanding of expectations is not. (...)

---

Acad Med. 2006 Aug;81(8):759-65.

A conceptual model for program evaluation in graduate medical education.

Musick DW.

Author information

Abstract

The author provides (1) a brief overview of the literature concerning program evaluation as applied to medical education, (2) a task-oriented conceptual model for use by residency directors in planning for program evaluation of graduate medical education training programs, (3) an explanation of the term "outcomes evaluation" including distinguishing between types of educational outcomes, and (4) a description of a five-step process of implementing the conceptual model.Recent accreditation standards for graduate medical education programs require a shift from a process-oriented to an outcomes-oriented model of evaluation. Accordingly, residency program directors must ensure compliance by undertaking comprehensive program evaluation procedures that demonstrate educational outcomes. Such procedures include attention to the need and focus of the evaluation; the evaluation methods to be used; and the documentation and presentation of evaluation results to key constituents. Involving teaching faculty and residents in developing a comprehensive evaluation program is vital to success. Regardless of philosophic debates pertaining to the appropriateness of the outcomes model for medical education, this approach appears likely to predominate in the foreseeable future particularly as related to the six general competencies of the physician. A practical, task-oriented approach will assist program directors in ensuring compliance with program evaluation standards.

PMID:

 

16868435

 

[PubMed - indexed for MEDLINE]