교육프로그램 평가를 위한 열두가지 팁(Med Teach, 2010)

교육프로그램 평가를 위한 열두가지 팁(Med Teach, 2010)

Twelve tips for evaluating educational programs

DAVID A. COOK

Mayo Clinic College of Medicine, USA

Introduction

For the past 3 months, Judy and John have been planning a workshop to teach postgraduate physician trainees how to examine the thyroid gland. They have spent countless hours studying different examination techniques, searching the literature for similar courses, patients gleaning teaching tips from colleagues, lining up with thyroid abnormalities,arranging for rooms and refreshments, and inviting trainees to attend. At about 3:00 p.m. the day before the workshop,John calls Judy and asks, ‘How are we going to know if this workshop is any good? How will we evaluate this program?’There is a moment of silence on the line, and then Judy replies,‘That’s a good question. I don’t know. I don’t even know where to start.’ 

Sound familiar? I hope not – but it is something I have seen too often. At one time or another, nearly all educators will need to evaluate an educational program. Why? Because we want to know the value of the activity into which we have invested time, energy, and other resources. As one classic text states, ‘Many different uses may be made of those value judgments . . . but the central purpose of the evaluative act is the same: to determine the merit or worth of some thing.’ (Worthen et al. 1997, p 8) Yet, as attention focuses on effective program development and implementation, the program evaluation may get neglected. Alternatively or additionally, an outstanding teacher might not possess the skills to conduct an effective evaluation.

누구의 의견이 중요한가?

Tip 1. First ask, ‘Whose opinion matters?’

The most important step in planning your evaluation is to identify for whom the information is intended (Figure 1). Who will read the final report? An evaluation intended for the medical school dean will look very different than one intended for publication in a peer reviewed journal. An evaluation intended to help yourself improve a course for the next go-round will require very different information than that in a final report to a funding agency demonstrating your program’s success. 

Stakeholders – people with an interest in the program and its evaluation – might include administrators, students, teachers, secretaries, fund-ing agencies, and the educational community at large.

그 사람에게 의미가 있는 정보는 무엇인가?

Tip 2. Next ask, ‘What would really be meaningful to them?’ 

The second most important step is to determine what would really be meaningful to your audience Different types and quantities of information will be more or less valuable in different situations. For example,although learner knowledge is often considered more important than satisfaction, in evaluating a new program to orient students to medical school, satisfaction may be more meaningful than performance on the end-of-year compre-hensive exam. 

어떻게 평가 결과가 사용될 것인가

An important factor in this decision is how the evaluation will be used. Evaluations are generally used to inform policy and guide decisions, such as

. Determining effectiveness, 

. Identifying areas for improvement, 

. Optimizing resource allocation, or 

. Empowering individuals (teachers, students, administrators, policy makers, etc) in their respective roles.

It is also helpful at this stage to consider whether you need a summative evaluation, a formative evaluation, or both. 

• Summative evaluations typically come at or near the end of a program or course. The intent is to inform a final pronouncement on the course: did it work or not? 

• Formative evaluations, on the other hand, seek to identify areas of strength and weakness so that a course can be improved. 

• Formative feedback is usually an ongoing process, whereas 

• summative feedback typically takes place at a single time point. 

Of course, the two are not mutually exclusive – you can do both if needed (but see Tip 12).

(학생)평가와 (프로그램)평가를 혼동하지 말기

Tip 3. Do not confuse evaluation with assessment

Educators commonly use the terms evaluation and assessment interchangeably. However, I find it useful to distinguish on assessment, evaluation, which focuses programs, from which focuses on learners (see Wilkes & Bligh 1999). Simply put, you assess learners to determine how well the learner is doing, and you evaluate programs to determine their merit or worth.

다른 사람들로부터 도움 받기

Tip 4. Get input from others

The adage ‘two heads are better than one’ is true in planning and conducting an evaluation. Seek input, not only from other educators, but also from other stakeholders such as students and administrators. Both those who will use the evaluation and those who are providing data (e.g., students and teachers) can offer suggestions on what they might find important.

다양한 패러다임.

Tip 5. Consider various evaluation paradigms and approaches

Objectives-oriented

First, and probably the best known to educators today, is the objectives-oriented approach. In this method you define the instructional goals or objectives at the start of the activity, and then at the end you evaluate to determine if these goals have been met. The specific outcome(s) studied depends on the objectives (outcomes will be discussed next). 

The strength of this approach lies in its simplicity – it facilitates a relatively uncomplicated design and straightforward interpretation of results. 

However, the objectives-oriented approach has several disadvantages. 

• First, it promotes tunnel vision and, by focusing on predetermined objectives, tends to be rather inflexible. It is poorly suited to capturing developments that arise unexpect- edly during implementation, and the evaluator can wind up a ‘slave’ to the objectives. 

• Second, if objectives are not carefully chosen the corresponding outcomes can potentially be trivial (‘learners will enjoy the course’) or infeasible (‘participants will become internationally-renowned experts in this topic’). 

• Third, educators may focus on achieving the outcomes themselves rather than facilitating lasting learning (teaching to the test), and they may inadvertently neglect other important teaching points.

Process-oriented

Next comes the process-oriented approach. In its most complete execution, this evaluation begins collecting data at the very inception of the idea for the program. It starts by determining if a need exists and if so what is the best way to meet that need. It then tracks the development process, monitors what actually happens during implementation, and typically concludes with a summative objectives-oriented evaluation at the end. 

The advantage in this approach is its comprehensiveness – providing information on each step in the program from start to finish. It thus provides both formative and summative information. 

단점 However, this comprehensive- ness comes at a price. It is very resource-intensive and complex, and generates voluminous data that may be difficult to interpret. It also requires tremendous foresight: once the program is underway, it is often too late to go back to the beginning and start collecting data.

Participant-oriented

Finally, we have the participant-oriented approach. This approach seeks to determine how the people involved perceived the program. It typically employs qualitative methods in which data collection and analysis follows an inductive and iterative process, with an ongoing cycle of data collection, data interpretation, recognition of need for specific additional data, and more data collection. Triangulation – the inclusion of data from multiple perspectives – is the key, and the evaluator will often solicit input from participants other than the learners themselves, such as teachers and support staff, as well as non-human sources such as the course syllabus or minutes of planning meetings. 

장점

This approach captures the complexity of a large program, including the local context, and includes a flexibility that enables it to respond readily to unintended effects. It also tends to be humanistic – focusing on the participants and their needs rather than intangible objectives and processes. 

단점

However, as with the process- oriented evaluation, this comes at the expense of cost and complexity. Also, both the data and its interpretation are highly subjective and strictly speaking apply only to the local context, which may bother some audiences.

성과를 먼저 정의하고, 그 다음 방법을, 그 다음 도구를

Tip 6. First select the outcome, then the measurement method, then the instrument, then the modality

When planning an evaluation, resist the tendency to start by selecting a specific instrument or tool (such as ‘licensure exam scores’ or ‘Mini-CEX’ or ‘SurveyMonkey’). Rather, first identify the outcome(s) you feel will be meaningful, then select the measurement method, then the instrument, and finally, the modality. I will discuss each of these in turn.

Outcomes are conceptual and often intangible constructs – things like ‘knowledge,’ ‘communication skill,’ ‘patient satis- faction,’ and ‘mortality.’ Kirkpatrick (1996) identified four broad classes of program outcomes: first Concentrating on the outcome at a conceptual level specific focus rather than jumping to a instrument helps attention on what would be most meaningful rather than what is convenient or familiar. For example, if communication skill will provide the most meaningful information to evaluate my course, then I should figure out how to assess communication skill rather than administer a written multiple-choice test

There are almost always multiple methods to measure a given outcome. For example, to assess knowledge, one could use self-report, a multiple-choice test, or faculty judgment. Each of these approaches has strengths and weaknesses, and each could be implemented in various ways.

We refer to specific implementations of a given method as instruments. Specific multiple-choice tests include the United States Medical Licensing Examination Step 1, an end-of-year cumulative exam, and a self-assessment quiz. Using assess- ment of clinical skills as another example, one method is direct observation of behavior, and one specific instrument using this method is the American Board of Internal Medicine’s Mini-CEX (Norcini et al. 2003).

Finally, many instruments can be implemented using the different modalities. For example, Mini-CEX can be implemented on paper, on the Internet, or on a personal digital assistance (PDA). A course evaluation survey could be administered on paper, via e-mail, by telephone (not com- monly done, but possible), or using an Internet-based tool such as SurveyMonkey (www.surveymonkey.com).

다양한 성과를 고려해보라 (그리고 측정방법과 측정도구도...)

Tip 7. Consider many different outcomes (and measures and instruments and modalities)

Before selecting the outcome(s) for your evaluation, spend some time thinking about alternatives.

교육목적과 부합하는 성과를 선택하라

Tip 8. Select outcomes that align with educational goals

It should go without saying that the evaluation outcomes should align with the educational goals, but unfortunately this is not always the case. For example, I have seen an evaluation in which the educational goal was to enhance learners’ communication skills, but the outcome assessed was actually knowledge about effective communication techniques. This knowledge might translate into enhanced communication, but there is no guarantee.

환자-관련 성과가 부적합한 세 가지 이유

However, I see at least three reasons not to treat patient-related outcomes as the holy grail of educational program evaluation. 

• First is the risk of misdirected emphasis. For example, if a course’s educational goal is to improve knowledge, a focus on patient-care outcomes might cause educators to emphasize algorithms that improve the measured outcome rather than facilitating deep understanding of underlying principles. 

• Second, since some patient-related outcomes are very difficult to assess, there is the risk of selecting an outcome measure (e.g., hemoglobin A1c levels) because it is measurable rather than because it is most important. 

• Third, measuring patient-related outcomes is simply not feasible in many instances. Issues such as statistical power (sample size), outcome sensitivity to change, dilution of effect (students’ decisions will be diluted as supervising physicians, other team members, health systems, and patient preferences come to bear (Shea 2001)),

신뢰도와 타당도를 고려하라

Tip 9. Consider the validity and reliability (or trustworthiness) of instrument scores 

Once you select the outcome and the method, you will begin to consider various instruments. It is best to start with instruments that already exist, not only because it can save you the work of developing an instrument de novo, but because published instruments usually have accrued evidence to support the validity of the information.

Regardless of the origin of the instrument, when it comes time to interpret the data, you will need to know how well you can trust the results. This requires evidence to support the validity of your interpretations (Cook & Beckman 2006). Note that we do not speak of the validity of the instrument itself, but the validity of interpretations (Downing 2003). 

The same instrument, applied to different uses, may provide for more or less valid interpretations. Moreover, validity is not a yes/no variable – it is a matter of degree. Higher-stakes evaluations will require greater validity evidence. Validity evidence is often available (published, or unpublished from authors) for existing instruments. However, you should still collect fresh evidence to support the validity of inferences in your educational context. 

파일럿테스트를 해보라

Tip 10. Pilot test the evaluation process 

Once you have identified or created your instrument(s), take time to pilot test the instrument and the data collection process prior to full-scale implementation. This allows you to detect and correct poorly worded questions, suboptimal formatting,and problems in administration and collection.

Tip 11. Obtain a sufficiently large and representative sample

The saying goes, ‘If you’ve seen one case of chest pain, you’ve seen one case of chest pain.’ Being able to identify myocardial infarction in high-risk men does not guarantee that I would recognize ischemia in a moderate-risk woman, or pulmonary embolism or gastroesophageal reflux. The concept – known in many fields as content specificity (Norman 2008) – that it often takes more than one question (frequently many more) to appropriately assess a learner or evaluate a course holds true across most outcomes, methods, and instruments. Even using qualitative methods, a single question will rarely suffice.

Obtain enough information to see a clear picture of what you are trying to evaluate. The instrument should be as long as needed – but no longer. 

Similarly, when using evaluation surveys or assessments of learning, be sure you have collected data from enough individuals. Sampling methods and sample size will depend on the evaluation design. 

• Quantitative outcomes typically involve statistical tests of inference, and sample size can be calculated using procedures available in standard texts.

• Qualitative studies might intentionally select participants to provide contrasting perspectives (purposive sampling), and continue obtaining information until no new themes emerge(saturation). 

Tip 12. Plan ahead and be realistic(you can not have it all) 

You have realized by now that a lot of work goes into planning an evaluation. Yet it is better to invest energy up front, and save yourself the headache and frustration that comes when a poorly planned evaluation fails to provide the information required to satisfy the audience and facilitate necessary decisions. 

One of the most difficult tasks in the planning process is determining where to draw the line. As the plan evolves, the list of desired information tends to grow longer and longer. It could easily reach the point that data collection instruments(e.g., questionnaires) exceed a reasonable length, or that demands on time and other support surpass available resources. You will have to make choices that retain the highest quality and most important (most meaningful) data.

---

Med Teach. 2010;32(4):296-301. doi: 10.3109/01421590903480121.

Twelve tips for evaluating educational programs.

Cook DA1.

Author information

• 1Mayo Clinic College of Medicine, Rochester, MN 55905, USA. cook.david33@mayo.edu

Abstract

At one time or another, nearly all educators will need to evaluate an educational program to determine its merit or worth. These tipswill help readers collect information to inform a meaningful evaluation, whether for local use or broad dissemination (i.e., research). The two most important questions in any evaluation are, 'Whose opinion matters?' and 'What would really be meaningful to them?' Other key steps include getting input from others, focusing on desired outcomes before selecting instruments, considering the validity or trustworthiness of the data, and pilot testing the evaluation process.

PMID:

 

20353325

 

DOI:

 

10.3109/01421590903480121