Likert 스케일, 측정의 수준과 통계의 '법칙' (Adv in Health Sci Educ, 2010)

Likert scales, levels of measurement and the ‘‘laws’’ of statistics

Geoff Norman

종종 통계기법을 걸고 넘어지는 리뷰어로부터 좌절을 겪는다.

One recurrent frustration in conducting research in health sciences is dealing with the reviewer who decides to take issue with the statistical methods employed.

그러나 일부 코멘트는 그저 틀린 말일 뿐이며, 연구설계의 문제보다는 리뷰어의 역량의 문제를 보여주곤 한다.

Some of these comments, like the proscription on the use of ANOVA with small samples, the suggestion to use power analysis to determine if sample size was large enough to do a parametric test, or the concern that a significant result still might be a Type II error, are simply wrong and reveal more about the reviewer’s competence than the study design.

여러가지가 지적되곤 하지만, 여전히 남는 문제는 '틀린 결론을 내릴 확률이 정말 높아지는가'에 대한 문제이다. 통계학자들은 이를 robustness라 부른다. 그 확률이 그다지 높아지지 않는다면, 진행해도 되는 것이다.

But what is left unsaid is how much it increases the chance of an erroneous conclusion. This is what statisticians call ‘‘robustness’’, the extent to which the test will give the right answer even when assumptions are violated. And if it doesn’t increase the chance very much (or not at all), then we can press on.

모수적 방법은 매우 다재다능하고 강력하다. 현대의 모수적 통계법은 정규분포의 간격척도 자료 사용을 가정하고 있다. 유사하게 일반화가능도이론도 ANOVA를 기반으로 한 것으로서 모수적 방법이다.

It is critically important to take this next step, not simply because we want to avoid ‘‘coming to the wrong conclusion’’. As it turns out, parametric methods are incredibly versatile, powerful and comprehensive. Modern parametric statistical methods like factor analysis, hierarchical linear models, structural equation models are all based on an assumption of normally distributed, interval-level data. Similarly generalizability theory, is based on ANOVA that again is a parametric procedure.

하나씩 알아보겠다.

I will explore the impact of three characteristics-sample size, non-normality, and ordinal-level measurement, on the use of parametric methods. The arguments and responses:

1) 샘플크기가 너무 작아서 모수적 방법을 사용할 수 없습니다.

1) You can’t use parametric tests in this study because the sample size is too small

어디에도 모수적 방법을 사용하는 가정으로 샘플사이즈에 대한 제한을 두고 있지는 않다. ANOVA와 t-test 는 동일한 가정을 기반으로 한다. 두 그룹에 대한 ANOVA의 F test는 t-test의 제곱과 같다. 어디에도 샘플사이즈가 작으면 비모수적방법이 모수적방법이 더 적절하다는 근거는 없다.

This is the easiest argument to counter. The issue is not discussed in the statistics literature, and does not appear in statistics books, for one simple reason. Nowhere in the assumptions of parametric statistics is there any restriction on sample size. It is simply not true, for example, that ANOVA can only be used for large samples, and one should use a t test for smaller samples. ANOVA and t tests are based on the same assumptions; for two groups the F test from the ANOVA is the square of the t test. Nor is it the case that below some magical sample size, one should use non-parametric statistics. Nowhere is there any evidence that non-parametric tests are more appropriate than parametric tests when sample sizes get smaller.

비모수적 방법이 극도록 보수적인(즉 틀린) 답을 내놓을 한 가지 상황이 있는데, 바로 자료를 이분화하는 것이다. 자료를 이분화하면 통계적 power를 크게 떨어뜨릴 수 있다.

In fact, there is one circumstance where non-parametric tests will give an answer that can be extremely conservative (i.e. wrong). The act of dichotomizing data (for example, using final exam scores to create Pass and Fail groups and analyzing failure rates, instead of simply analyzing the actual scores), can reduce statistical power enormously.

샘플사이즈가 안 중요한 것은 아니다. 몇 가지 이유로 인해서 문제가 될 수 있다.

Sample size is not unimportant. It may be an issue in the use of statistics for a number of reasons unrelated to the choice of test:

(a) 너무 샘플 수가 작으면 외적타당도가 문제가 될 수 있다.

(a) With too small a sample, external validity is a concern. It is difficult to argue that 2 physicians or 3 nursing students are representative of anything (qualitative research notwithstanding). But this is an issue of judgment, not statistics.

(b) 샘플 수가 작으면 분포에 대한 우려가 있다. 하지만 5명 이상이면 족하다. 걱정해야 할 것은 검사를 수행할 수 있느냐가 아니라 검사의 robustness이다.

(b) As we will see in the next section, when the sample size is small, there may be concern about the distributions (see next section). However, it turns out that the demarcation is about 5 per group. And the issue is not that one cannot do the test, but rather that one might begin to worry about the robustness of the test.

(c) 샘플 수가 작으면 더 큰 효과가 있어야만 통계적 유의성이 나타난다. 그러나 '통계적으로 유의한 것은 통계적으로 유의한 것이다'

(c) Of course, small samples require larger effects to achieve statistical significance. But to say, as one reviewer said above, ‘‘Given the small number of participants in each group, can the authors claim statistical significance?’’, simply reveals a lack of understanding. If it’s significant, it’s significant. A small sample size makes the hurdle higher, but if you’ve cleared it, you’re there.

2) 데이터가 정규분포를 따르지 않기 때문에 t test나 ANOVA를 사용할 수 없다.

2) You can’t use t tests and ANOVA because the data are not normally distributed

This is likely one of the most prevalent myths. We all see the pretty bell curves used to illustrate z tests, t tests and the like in statistics books, and we learn that ‘‘parametric tests are based on the assumption of normality’’. Regrettably, we forget the last part of the sentence. For the standard t tests ANOVAs, and so on, it is the assumption of normality of

the distribution of means, not of the data. The Central Limit Theorem shows that, for sample sizes greater than 5 or 10 per group, the means are approximately normally distributed regardless of the original distribution. Empirical studies of robustness of ANOVA date all the way back to Pearson (1931) who found ANOVA was robust for highly skewed non-normal distributions and sample sizes of 4, 5 and 10. Boneau (1960) looked at normal, rectangular and exponential distributions and sample sizes of 5 and 15, and showed that 17 of the 20 calculated P-values were between .04 and .07 for a nominal 0.05. Thus both theory and data converge on the conclusion that parametric methods examining differences between means, for sample sizes greater than 5, do not require the assumption of normality, and will yield nearly correct answers even for manifestly nonnormal and asymmetric distributions like exponentials.

3) ANOVA나 Pearson correlation (혹은 회귀분석)과 같은 모수적 방법은 자료가 서열척도고 그래서 정규성을 가정할 수 없다면 사용해서는 안된다.

3) You can’t use parametric tests like ANOVA and Pearson correlations (or regression, which amounts to the same thing) because the data are ordinal and you can’t assume normality.

개별 Likert 척도가 서열척도라도, 여러 Likert 척도의 합은 등간척도이다.

The question, then, is how robust are Likert scales to departures from linear, normal distributions. There are actually three answers. The first, perhaps the least radical, is that (...) But their strongest argument appears to be that while Likert questions or items may well be ordinal, Likert scales, consisting of sums across many items, will be interval.

숫자는 숫자일 뿐이다.

The second approach, as elaborated by Gaito (1980), is that this is not a statistics question at all. The numbers ‘‘don’t know where they came from’’.

컴퓨터는 주어진 숫자에 대한 결론을 주는 것일 뿐이다. 그 숫자에 대한 결론이 틀렸다고 할 수는 없드며, 연구자가 결정해야 할 것은 그 숫자에 대해서 이뤄진 분석이 그 아래 깔려있는 구인(construct)를 잘 반영하는가에 대한 판단이다.

And all the computer can do is draw conclusions about the numbers themselves. So if the numbers are reasonably distributed, we can make inferences about their means, differences or whatever. We cannot, strictly speaking, make further inferences about differences in the underlying, latent, characteristic reflected in the Likert numbers, but this does not invalidate conclusions about the numbers. This is almost a ‘‘reductio ad absurbum’’ argument, and appears to solve the problem by making it someone else’s, but not the statistician’s problem. After all, someone has to decide whether the analysis done on the numbers reflects the underlying constructs, and Gaito provides no support for this inference.

ANOVA나 다른 비슷한 검사에 대해서는 비정규성으로 인해서 생길 수 있는 문제는 앞에서 다뤘다.

So let us return to the more empirical approach that has been used to investigate robustness. As we showed earlier, ANOVA and other tests of central tendency are highly robust to things like skewness and non-normality. Since an ordinal distribution amounts to some kind of nonlinear relation between the number and the latent variable, then in my view the answer to the question of robustness with respect to ordinality is essentially answered by the studies cited above showing robustness with respect to non-normality. 

그러나 상관관계나 회귀분석은 어떨까? 여기서는 더 이상 평균의 분포에 대해서 이야기하지 않는다. 여기서는 분포의 양 극단이 어딘가 - 궁극적으로 회귀선을 고정(anchor)하게 되므로 - 가 중요하다. 따라서 skewness나 비정규성이 틀린 답을 줄 수도 있다.

However, when it comes to correlation and regression, this proscription cannot be dealt with quite so easily. The nature of regression and correlation methods is that they inherently deal with variation, not central tendency (Cronbach 1957). We are no longer talking about a distribution of means. Rather, the magnitude of the correlation is sensitive to individual data at the extremes of the distribution, as these ‘‘anchor’’ the regression line. So, conceivably, distortions in the distribution—skewness or non-linearity—could well ‘‘give the wrong answer’’.

만약 Likert 분포가 왜곡되어있거나 다른 바람직하지 못한 특성을 가진다면 상관관계나 회귀계수를 산출할지 말지가 통계적 문제가 되고, 이는 다시 'robustness'의 문제라고 할 수 잇다. 여기서는 중심극한정리가 도움이 되지는 않지만, 우리에게 도움이 될 만한 연구들이 있다. Pearson correlation은 왜도나 비정규성에 대해서 robust하다.

If the Likert ratings are ordinal which in turn means that the distributions are highly skewed or have some other undesirable property, then it is a statistical issue about whether or not we can go ahead and calculate correlations or regression coefficients. It again becomes an issue of robustness. If the distributions are not normal and linear. what happens to the correlations? This time, there is no ‘‘Central Limit Theorem’’ to provide theoretical confidence. However, there have been a number of studies that are reassuring. Pearson (1931, 1932a, b), Dunlap (1931) and Havlicek and Peterson (1976) have all shown, using theoretical distributions, that the Pearson correlation is robust with respect to skewness and nonnormality. (...)They concluded that ‘‘The Pearson r is rather insensitive to extreme violations of the basic assumptions of normality and the type of scale’’.

Spearman과 Pearson 계수의 상관관계는 0.99이고 기울기는 1.001이었다. 심각하게 왜곡된 자료에서도 비슷했다. 둘은 거의 동일한 결과를 주는 것이다. 동순위자가 많은 경우에 Spearman이 조금 다른 답을 주긴 하지만, 이는 Spearman이 동점자를 처리하는 방식의 문제이지 Pearson 상관의 문제는 아니다. Pearson correlation은 이들 가정에 위배되더라도 매우 robust하다.

For the original data, the correlation between Spearman and Pearson coefficients was 0.99, the slope was 1.001, and the intercept was -.007. Even with the severely skewed data, the correlation was still 0.987, the slope was 0.995, and the intercept was -.0003. The means of the Pearson and Spearman correlations were within 0.004 for all conditions. For this set of observations, the Pearson correlation and the Spearman correlation based on ranks yielded virtually identical values, even in conditions of manifestly non-normal, skewed data. Now it turns out that, when you have many tied ranks, the Spearman gives slightly different answers than the Pearson, but this reflects error in the Spearman way of dealing with ties, not a problem with the Pearson correlation. The Pearson correlation like all parametric tests we have examined, is extremely robust with respect to violations of assumptions.

4) 명목, 순위 척도에서는 ICC(혹은 일반화가능도 이론)을 사용할 수 없으며 Kappa나 Weighted Kappa를 사용해야 한다.

4) You cannot use an intraclass correlation (or Generalizability Theory) to compute the reliability because the data are nominal/ordinal and you have to use Kappa (or Weighted Kappa)

Kappa was originally developed as a ‘‘Coefficient of agreement for nominal scales’’ (Cohen 1960), and in its original form was based on agreement expressed in a 2 9 2 frequency table. Cohen (1968) later generalized the formulation to ‘‘weighted kappa’’, to be used with ordinal data such as Likert scales, where the data would be displayed as agreement in a 7 9 7 matrix. Weighting accounted for partial agreement (Observer 1 rates it 6; Observer 2 rates it 5). Although any weighting scheme is possible, the most common is ‘‘quadratic’’ weights, where disagreement of 1 unit is weighted 1, of 2 is weighted 4, of 3, 9, and so forth.

Surprisingly, if one proceeds to calculate an intraclass correlation with the same 7-point scale data, the results are mathematically identical, as proven by Fleiss and Cohen (1973). And if one computes an intraclass correlation from a 2 9 2 table, using ‘‘1’’ when there is agreement and ‘‘0’’ when there is not, the unweighted kappa is identical to an ICC. Since ICCs and G theory are much more versatile (Berk 1979), handling multiple observers and multiple factors with ease this equivalence is very useful.

Summary

Parametric statistics can be used with Likert data, with small sample sizes, with unequal variances, and with non-normal distributions, with no fear of ‘‘coming to the wrong conclusion’’. These findings are consistent with empirical literature dating back nearly 80 years. The controversy can cease (but likely won’t).

Adv Health Sci Educ Theory Pract. 2010 Dec;15(5):625-32. doi: 10.1007/s10459-010-9222-y. Epub 2010 Feb 10.

Likert scales, levels of measurement and the "laws" of statistics.

의학교육에서의 평가: 일반화가능도 이론의 개념

Medical education assessment: a brief overview of concepts in generalizability theory

Mohsen Tavakol1, Robert L. Brennan2

1Medical Education Unit, The University of Nottingham, UK

2Centre for Advanced Studies in Measurement and Assessment, The University of Iowa, USA

의학교육자들은 학생평가의 질 향상을 위해서 측정오차가 발생하는 원인을 알아야 한다.

General Medical Council (GMC) in the UK has emphasized the importance of internal consistency for students’ assess-ment scores in medical education.1 Typically Cronbach’s alpha is reported by medical educators as an index of internal consistency. Medical educators mark assessment questions and then estimate statistics that quantify the consistency (and, if possible, the accuracy and appropriate-ness) of the assessment scores in order to improve subse-quent assessments. The basic reason for doing so is the recognition that student marks are affected by various types of errors of measurement which always exist in student marks, and which reduce the accuracy of measurement. The magnitude of measurement errors is incorporated in the concept of reliability of test scores, where reliability itself quantifies the consistency of scores over replications of a measurement procedure. Therefore, medical educators need to identify and estimate sources of measurement error in order to improve students’ assessment.

CTT에서 학생의 진점수는 관찰점수와 하나의 미분화된 오차항의 합이다. 이 모델에서 흔히 사용되는 신뢰도 척도는 Cronbach's alpha이다. 그러나 언제나 그랬든 alpha는 item들의 표본과 관련된 오차만 포함되어있다고 볼 수 있다. 따라서 우리는 alpha로부터 서로 다른 측정의 소스가 유발하는 오차에 대한 영향력을 집어내거나 고립시키거나 추정할 수 없다. CTT의 확장된 것이 G이론이며, 이는 'facet'이라 불리는 다양한 측정오차의 원인을 구분할 수 있게 해준다. 

Under the Classical Test Theory (CTT) model, the stu-dent’s true score is the sum of the student’s observed score and a single undifferentiated error term. Using this model, the most frequently reported estimate of reliability is Cronbach’s alpha. Almost always, however, when alpha is reported, it incorporates errors associated with sampling of items, only. Accordingly, alpha does not allow us to pin-point, isolate, and estimate the impact of different sources of measurement error associated with observed student marks. An extension of CTT called “G (Generalizability) theory” enables us to differentiate the multiple, potential sources of measurement error called “facets” (sometimes called “dimensions” in experimental design literature). 

모든 facet의 집합은 인정가능한 관측(admissible observation)의 모든 측면(universe)라고 할 수 있다.

For exam-ple, in an OSCE exam, a student might be observed by one of a large sample of examiners, for one of a large sample of standardized patients (SPs), and for one of a large sample of cases. The facets, then, would be examiners, SPs and cases---each of which serves as a potential source of measurement error. The set of all facets constitutes the universe of admissible observations (UAO) in the terminology of G theory. As another example, suppose that for a cardiology exam, the investigator is interested in an item facet, only; in that case, there is only one facet.

어떤 facet을 사용해야 하는지, 얼마나 많은 facet을 사용해야 하는가에 대한 정답은 없다. 이를 결정하는 것은 연구자의 책임이며, 각 facet의 중요도에 대한 근거를 제시할 수 있어야 한다. 

There is no right answer to the question of which facets, or how many facets, should be included in the UAO. It is the investigator’s responsibility to justify any decision about the inclusion of facets, and provide supporting evidence about the importance of each facet to the consistency and accuracy of the measurement procedure. G theory provides a conceptual framework and statistical machinery to help an investigator do so.

특정 검사에는 각 facet별로 구체적인 조건들의 숫자가 정해져 있다. UG의 정의. (CTT의 진점수에 대응되는 것이다)

For any given form of a test, there are a specified num-ber of conditions for each facet. The (hypothetical) set of all forms similarly constructed is the called the universe of generalization (UG). For any given examinee, we can conceive of getting an average score over all such forms in the UG. This average score is called the student’s universe score, which is the analogue of true score in CTT. The variance of such universe scores, called universe score variance, can be estimated using the analysis of variance “machinery” employed by G theory.

G이론에서는 다양한 설계가 가능하다.

G theory can accommodate numerous designs to exam-ine the measurement characteristics of many kinds of student assessments. If medical educators wish to investi-gate assessment items as a single source of measurement error on a test, this is a single facet design. There are two types of single-facet designs. If the same sample of questions is administered to a cohort of students, we say the design is crossed in that all students (s) respond to all items (i). This crossed design is symbolised as s × i, and read students are crossed within items. If each student takes a different set of items, we have a nested design, which is symbolised i:s meaning that items are nested within students.

In most realistic circumstances there are facets in addi-tion to items. Imagine a case-based assessment with four cases and a total of 40 items designed to measure the ability of students about dermatology. In this example, all students take all items; hence, students are crossed within items (s × i), but items are distributed into cases (e.g., 10 items in case 1, 10 items in case 2, 10 items in case 3 and 10 items in case 4). That is, items are nested within cases, and this design is called a two-facet nested design that is symbolised as s × (i:c).

특정 facet에 대해서 variance component가 크다면, 이 facet이 학생 점수에 상대적으로 큰 영향을 줬다는 의미이다. 예를 들어 OSCE에서 만약 시험관(examiner)에 대한 variance component가 높게 나왔다면, 시험관이 평가에 있어서 일관되게 행동하지 못했음을 보여주는 것이다.

The designs discussed in the previous paragraphs are usually called G study designs, and they are associated with the UAO. The principal purpose of such designs is to collect data that can be used to estimate what are called “variance components.” In essence, the set of variance components for the UAO provides a decomposition of the total observed variance into its component parts. These component parts reflect the differential contribution of the various facets; i.e., a relatively large variance component associated with a facet indicates that the facet has a relatively large impact on student marks. For example, in an OSCE, if the variance component for examiners (the examiner facet) is estimated as high, we would conclude that the examiners have not behaved consistently in their rating of the construct of interest.

variance component를 계산하고 나면, 연구자들은 error variance를 추정하고 UG와 관련된 reliability-like coefficient를 를 계산한다. 

Once variance components are estimated, typically in-vestigators estimate error variances and reliability-like coefficients that are associated with the UG. Such coeffi-cients can range from 0 to 1. 

One coefficient is called a generalizability coefficient; it incorporates relative error variance. Another coefficient is called a Phi coefficient; it incorporates absolute error variance. 

Computing these coefficients and error variances requires specifying the D study design which, in turn, specifies the number of condi-tions of each facet that are (or will be) used in the opera-tional measurement procedure. 

Relative error variance (and, hence, a generalizability coefficient) is appropriate when interest focuses on the rank ordering of students. 

Absolute error variance (and, hence, a Phi coefficient) is appropriate when interest focuses on the actual or “abso-lute” scores of students. 

Relative error variance (for a so-called “random effects” model) involves all the variance components that are interactions between students and facets. 

Absolute error variance includes relative error variance plus the variance components for the facets them-selves. The square root of these error variances are called standard errors of measurement. They can be used to establish confidence intervals for students’ universe scores. For further information about the these coefficients and error variances, readers may refer to particular books.2,3

Knowing the magnitude of estimated variance compo-nents enables us to design student assessments that are optimal, at least from a measurement perspective. For example, a relatively small estimated variance component for the interaction of students and items suggests that a relatively small number of items may be sufficient for a test to achieve an acceptable level for a generalizability coeffi-cient.

In practice, powerful computer programs are required to estimate variance components, coefficients, and error variances, especially for multifaceted designs. Several G theory software programs have been developed for estimating such statistics (see, for example, http://www. education.uiowa.edu/centers/casma/computer-programs).

Variance components can also be estimated using SPSS and SAS, but these packages do not directly estimate coefficients and error variances. The first author is develop-ing an online user friendly application for estimating variance components, for both balanced and unbalanced designs. Using a simple script, readers will be able to print out the estimates of important parameters in G theory. The application is written in R and C++ languages and executed by PHP codes. Figure 1 shows a balanced design output from the application.

Medical education assessment: a brief overview of concepts in generalizability theory

Cronbach's alpha 이해하기 (IJME, 2011)

Making sense of Cronbach’s alpha

Mohsen Tavakol, Reg Dennick

International Journal of Medical Education

Reliable하지 않은 도구는 valid할 수 없다.

An instrument cannot be valid unless it is reliable. However, the reliability of an instrument does not depend on its validity.2

What is Cronbach alpha?

1951년 개발되었음. 0과 1사이의 값을 가진다.

Alpha was developed by Lee Cronbach in 195111 to provide a measure of the internal consistency of a test or scale; it is expressed as a number between 0 and 1.

내적일관성은 검사를 수행하기 전에 결정되어야 한다. 신뢰도는 한 검사에서 측정오차의 양이 어느정도인지를 보여주는 것이다. 상관관계를 제곱해서 1에서 빼면 측정오차의 index가 된다. 즉, 신뢰도가 0.80이라면 무작위오차가 0.36이라는 의미이다.

Internal consistency should be determined before a test can be employed for research or examination purposes to ensure validity. In addition, reliability estimates show the amount of measurement error in a test. Put simply, this interpretation of reliability is the correlation of test with itself. Squaring this correlation and subtracting from 1.00 produces the index of measurement error. For example, if a test has a reliability of 0.80, there is 0.36 error variance (random error) in the scores (0.80×0.80 = 0.64; 1.00 – 0.64 = 0.36).12

검사에 포함된 문항이 서로 연관되어있다면, alpha는 올라간다. 그러나 alpha가 높은 것이 언제나 높은 내적일관성을 보장하는 것은 아니다. 왜냐면 이는 alpha가 검사지의 길이에 의해서도 영향을 받기 때문이다. 검사 문항이 너무 적으면 alpha는 떨어진다. 따라서 alpha를 높이기 위해서 더 많은 문항을 포함시킬 수 있다. 또한 alpha는 특정 검사자 표본에 대한 것이다. 따라서 다른 문헌에서 제시된 alpha에만 의존해서는 안되며 검사를 하면 alpha를 매번 새로 구해야 한다.

If the items in a test are correlated to each other, the value of alpha is increased. However, a high coefficient alpha does not always mean a high degree of internal consistency. This is because alpha is also affected by the length of the test. If the test length is too short, the value of alpha is reduced.2, 14 Thus, to increase alpha, more related items testing the same concept should be added to the test. It is also important to note that alpha is a property of the scores on a test from a specific sample of testees. Therefore investigators should not rely on published alpha estimates and should measure alpha each time the test is administered. 14

homogeneity이 '단차원성'을 의미하는 것과 비교해서, 내적일관성은 문항들의 상호연관성과도 관련이 있다. 한 척도가 '단차원적'이라는 것은 모든 문항이 하나의 특성 혹은 구인을 측정한다는 의미이다. 내적일관성은 균일성/단차원성 측정의 필요조건이지만 충분조건은 아니다. 근본적으로 신뢰도라는 개념은 검사문항의 단차원성을 가정하고 있으며, 만약 이 가정이 위배된다면 신뢰도를 과소평가하는 주요 원인이 될 수 있다. 다차원성을 갖는 검사지가 단차원성을 갖는 검사지에 비해서 alpha가 반드시 낮지는 않다는 점은 잘 알려져 있다. 따라서 검사의 '내적일관성'에 대한 index라고 보는 것이 alpha를 좀더 정확히 해석하는 것이다.

Internal consistency is concerned with the interrelatedness of a sample of test items, whereas homogeneity refers to unidimensionality. A measure is said to be unidimensional if its items measure a single latent trait or construct. Internal consistency is a necessary but not sufficient condition for measuring homogeneity or unidimensionality in a sample of test items.5, 15 Fundamentally, the concept of reliability assumes that unidimensionality exists in a sample of test items16 and if this assumption is violated it does cause a major underestimate of reliability. It has been well documented that a multidimensional test does not necessary have a lower alpha than a unidimensional test. Thus a more rigorous view of alpha is that it cannot simply be interpreted as an index for the internal consistency of a test. 5, 15, 17

alpha는 단순이 검사 문항의 단차원성을 측정하는 것이 아니지만, 문항들이 단차원적인지 확인하는 용도로 활용될 수는 있다. 반대로 한 검사가 두 개 이상의 개념(구인)을 검사하고 있다면, 전체 검사지의 alpha를 보고하는 것은 문항의 숫자를 늘리는 것과 같아서 결과적으로 alpha를 inflation시키는 것이다. 따라서 원칙적으로 alpha는 각각의 개념에 대해서 계산되어야 한다. 비균질한, 다양한 사례 바탕으로 한 문항들로 구성된 총괄평가에서 alpha는 각 사례별로 계산되어야 한다.

Alpha, therefore, does not simply measure the unidimensionality of a set of items, but can be used to confirm whether or not a sample of items is actually unidimensional. 5 On the other hand if a test has more than one concept or construct, it may not make sense to report alpha for the test as a whole as the larger number of questions will inevitable inflate the value of alpha. In principle therefore, alpha should be calculated for each of the concepts rather than for the entire test or scale. 2, 3 The implication for a summative examination containing heterogeneous, casebased questions is that alpha should be calculated for each case.

더 중요한 것은 alpha가 'tau equivalent model'에 근거하고 있다는 것이다. 이 모델은 각각의 문항이 같은 scale로 같은 latent trait를 측정하고 있음을 가정한다. 따라서 factor analysis를 통해서 밝힐 수 있듯, 다수의 trait을 다루고 있다면 이 가정에 위배되는 것이며 alpha는 검사의 reliability를 과소평가하게 된다. 검사문항의 숫자가 너무 작다면 이 역시 tau-equivalence 가정을 위배하는 것이고, 신뢰도를 과소추정할 수 있다. 현실에서 alpha는 신뢰도의 하한추정값(lower-bound estimate)이며, 왜냐하면 비균질한 검사문항이 tau-equivalent model의 가정을 위배하는 것이기 때문이다. SPSS에서 'standardised item alpha'가 'Cronbach's alpha'보다 높다면 tau equivalent measurement에 대한 추가 검사가 필요하다.

More importantly, alpha is grounded in the ‘tau equivalent model’ which assumes that each test item measures the same latent trait on the same scale. Therefore, if multiple factors/traits underlie the items on a scale, as revealed by Factor Analysis, this assumption is violated and alpha underestimates the reliability of the test.17 If the number of test items is too small it will also violate the assumption of tau-equivalence and will underestimate reliability.20 When test items meet the assumptions of the tau-equivalent model, alpha approaches a better estimate of reliability. In practice, Cronbach’s alpha is a lower-bound estimate of reliability because heterogeneous test items would violate the assumptions of the tau-equivalent model.5 If the calculation of “standardised item alpha” in SPSS is higher than “Cronbach’s alpha”, a further examination of the tauequivalent measurement in the data may be essential.

Numerical values of alpha

검사문항의 숫자, 상호연관성, 차원성 등이 alpha값에 영향을 준다. 수용가능한 alpha값에 대한 보고는 0.7에서 0.95까지의 범위에 이른다. alpha값이 낮은 것은 문항 수가 작거나, 상호연관성이 적거나, 비균질한 구인때문일 수 있다. 만약 낮은 alpha값이 문항간 낮은 상관관계에 기인한 것이라면 일부 문항을 수정하거나 버려야 한다. 가장 쉬운 방법은 각 문항과 총점의 상관관계를 구해보는 것이다. 상관관계가 낮은 문항을 버리면 된다. 만약 alpha가 너무 높다면, 이는 일부 문항은 서로 다른 문항인 척 위장하고 있지만 사실상 같은 문제임을 보여주는 것이다. alpha의 최대값은 0.90정도가 추천된다.

As pointed out earlier, the number of test items, item interrelatedness and dimensionality affect the value of alpha.5 There are different reports about the acceptable values of alpha, ranging from 0.70 to 0.95. 2, 21, 22 A low value of alpha could be due to a low number of questions, poor interrelatedness between items or heterogeneous constructs. For example if a low alpha is due to poor correlation between items then some should be revised or discarded. The easiest method to find them is to compute the correlation of each test item with the total score test; items with low correlations (approaching zero) are deleted. If alpha is too high it may suggest that some items are redundant as they are testing the same question but in a different guise. A maximum alpha value of 0.90 has been recommended.14

Making sense of Cronbach's alpha

Programmatic Assessment를 위한 열두가지 팁(Medical Teacher, 2014)

12 Tips for programmatic assessment

C.P.M. VAN DER VLEUTEN1, L.W.T. SCHUWIRTH2, E.W. DRIESSEN1, M.J.B. GOVAERTS1 & S. HEENEMAN1

1Maastricht University, Maastricht, The Netherlands, 2Flinders University, Adelaide, Australia

Introduction

평가 프로그램을 구성할 때, 개별 평가는 '모든 평가의 합은 개별 평가의 단순합보다 크다'라는 생각을 가지고 선택되어야 한다. 따라서 개별 평가가 모두 완벽할 필요는 없다. 여러 평가법을 혼합한 결과가 이상적이어야 한다. 

From the notion that every individual assessment has severe limitations in any criterion of assessment quality (Van der Vleuten 1996), we proposed to optimise the assessment at the programme level (Van der Vleuten & Schuwirth 2005). In a programme of assessment, individual assessments are purposefully chosen in such a way that the whole is more than the sum of its parts. Not every individual assessment, therefore, needs to be perfect. The dependability and credibility of the overall decision relies on the combination of the emanating information and the rigour of the supporting organisational processes. Old methods and modern methods may be used, all depending on their function in the programme as a whole. The combination of methods should be optimal. After the introduction of assessment programmes we have published conceptual papers on it (Schuwirth & Van der Vleuten 2011, 2012) and a set of guidelines for the design of programmes of assessment (Dijkstra et al. 2012). More recently we proposed an integrated model for programmatic assessment that optimised both the learning function and the decision-making function in competency-based educational contexts (Van der Vleuten et al. 2012), using well-researched principles of assessment (Van der Vleuten et al. 2010). 

Dijkstra의 가이드라인이 보다 일반적이고 교육과정이 존재하지 않는 평가프로그램에도 적용가능하다고 한다면, 통합적 모델은 구성주의적 학습 프로그램 평가를 위한 것이다. PA(programmatic assessment) 에서의 의사결정은 개별 평가와 분리된다. 각각의 평가는 학습자에 대한 정보를 모으는 것이 목적이다. 이 때의 결정은 각각의 평가에서 충분한 정보가 수집되었을 때 내려진다. PA는 학습에 대한 종단적 관점을 포괄하며, 특정 학습성과와 관련해서 평가가 이뤄진다. 성장과 발달을 모니터하고 멘토링을 제공한다. 정보가 모두 모여졌을 때 서로 독립적인 평가자 그룹에 의해서 의사결정을 내린다. 이러한 PA모델은 교육에서 많이 인정되는 반면, 많은 사람들은 PA가 복잡하고 이론에 불과하다고 생각한다. 

Whereas the Dijkstra et al. guidelines are generic in nature and even apply to assessment programmes without a curriculum (e.g. certification programmes), the integrated model is specific to constructivist learning programmes. In programmatic assessment decisions are decoupled from individual assessment moments. These individual assessment moments primarily serve for gathering information on the learner. Decisions are only made when sufficient information is gathered across individual moments of assessment. Programmatic assessment also includes a longitudinal view of learning and assessment in relation to certain learning outcomes. Growth and development is monitored and mentored. Decision-making on aggregated information is done by an (independent) group of examiners. Although this model of programmatic assessment is well received in educational practice (Driessen et al. 2012; Bok et al. 2013), many find programmatic assessment complex and theoretical. Therefore, in this paper we will describe concrete tips to implement programmatic assessment.

평가를 위한 마스터플랜을 세우라

Tip 1 Develop a master plan for assessment

역량프레임워크 형태로 큰 틀에서의 구조를 선택해야 한다. 개별 평가에 대해서 모두 P/F 결정을 내리는 것이 아니라 다양한 평가가 이루어진 다음에 일관된 평가를 내려야 하기 때문이다. 기존의 형성평가와 종합평가라는 개념은 '저부담' 과'고부담' 의사결정으로 새롭게 정의된다. '고부담'결정은 많은 자료를 필요로 한다. 

Just like a modern curriculum is based on a master plan, programmatic assessment has to be based on such a master plan as well. Essential here is the choice for an overarching structure usually in the form of a competency framework. This is important since in programmatic assessment pass/fail decisions are not taken at the level of each individual assessment moment, but only after a coherent interpretation can be made across many assessment moments. An individual assessment can be considered as a single data point. The traditional dichotomy between formative and summative assessment is redefined as a continuum of stakes, ranging from low- to high-stakes decisions. The stakes of the decision and the richness of the information emanating from the data points are related, ensuring proportionality of the decisions: high-stake decisions require many data points. In order to meaningfully aggregate information across these data points an overarching structure is needed, such as a competency framework. Information from various data points can be combined to inform the progress on domains or roles in the framework. For example, information on communication from an objective structured Clinical examination (OSCE) may be aggregated with information on communication from several mini-clinical evaluation exercise (Mini-CEX) and a multisource-feedback tool.

따라서 마스터플랜은 전체 평가구조와 교육과정에서 각 데이터포인트가 어디에 위치하는지를 보여주는 지도가 되어야 한다. 실제 상황에서 이뤄지는 직접관찰과 같은 비표준화된 조건의 평가도 있고 이런 경우 전문가의 판단이 불가피하다. 학습 단계에 따라서 마스터플랜에는 표준화된 방법과 비표준화된 방법이 혼합된다. 교육과정에 대한 마스터플랜과 평가에 대한 마스터플랜은 이상적으로 하나의 마스터플랜이어야 한다.

The master plan should therefore also provide a mapping of data points to the overarching structure and to the curriculum. The choices for each method and its content are purposefully chosen with a clear educational justification for using this particular assessment in this part of the curriculum in this moment in time. Many competency frameworks emphasise complex skills (collaboration, professionalism, communication, etc.) that are essentially behavioural, and therefore require longitudinal development. They are assessed through direct observation in real-life settings, under unstandardised conditions, in which professional, expert judgement is imperative. Depending on the curriculum and the phase of study, the master plan will thus contain a variety of assessment contents, a mixture of standardised and non-standardised methods and the inclusion of modular as well as longitudinal assessment elements. For any choice, the contribution to the master plan and through this alignment with the curriculum and the intended learning processes is crucial. The master plan for the curriculum and the assessment is ideally one single master plan.

그 결과 비표준화된 평가의 주관성은 두 가지 측면에서 PA에 영향을 준다.

The resulting subjectivity from non-standardised assessment using professional judgement is something that can be dealt with in programmatic assessment in two ways. 

First, by sampling many contexts and assessors, because many subjective judgements provide a stable generalisation from the aggregated data (Van der Vleuten et al. 1991). 

Second, because subjectivity can be dealt through bias-reduction strategies showing due process in the way decisions are reached. We will revisit these latter strategies later in Tip 6. Subjectivity is not dealt with by removing professional judgement from the assessment process, for example, by over-structuring the assessment.

피드백을 장려하는 평가 규정을 개발하라

Tip 2 Develop examination regulations that promote feedback orientation

총괄평가식 접근에서 피드백은 대체로 잊혀지게 된다. 개별 평가와 credit을 연결시킬수록 학습자는 피드백을 받고 그것을 따르려 하기보다는 어떻게 시험에서 통과할지만 고민하게 된다. Credit point는 고부담 결정에만, 그리고 여러 데이터포인트를 기반으로 뒤따라야 한다. 

Individual data points are optimised for providing information and feedback to the learner about the quality of their learning and not for pass/fail decisions. Pass–fail decisions should not be made on the basis of individual data points – as is often the case in traditional regulations. Examination regulations traditionally connect credits to individual assessments; this should be prevented in programmatic assessment. Research has shown that feedback is ignored in assessment regimes with a summative orientation (Harrison et al. 2013). Because lining credits to individual assessments raises their stake, learners will primarily orientate themselves on passing the test instead of on feedback reception and follow-up (Bok et al. 2013). Credit points should be linked only to high stake decisions, based on many data points. In all communication and most certainly in examination regulations the low-stake nature of individual assessments should be given full reign.

정보 수집을 위한 견고한 시스템을 도입하라

Tip 3 Adopt a robust system for collecting information

e-portfolio는 다음과 같은 장점이 있다.

In programmatic assessment, information about the learner is essential and massive information is gathered over time. Being able to handle this information flexibly is vital. One way of collecting information is through the use of (electronic) portfolios. Here, portfolios have a dossier function allowing periodic analyses of the student’s competence development and learning goals. The (e-)portfolio should therefore serve three functions: 

• (1) provide a repository of formal and informal assessment feedback and other learning results (i.e. assessment feedback, activity reports, learning outcome products, and reflective reports), 
• (2) facilitate the administrative and logistical aspects of the assessment process (i.e. direct online loading of assessment and feedback forms via multiple platforms, regulation of who has access to which information and by connecting information pieces to the overarching framework), and 
• (3) enable a quick overview of aggregated information (such as overall feedback reports across sources of information). User friendliness is vital. The (e-)portfolio should be easily accessible to whatever stakeholder who has access to it. Many e-portfolios are commercially available, but care should be taken to ensure that the structure and functionalities of these portfolios are sufficiently aligned with the requirements of the assessment programme.

모든 저위험평가가 학습자들에게 의미있는 피드백을 제공하도록 하라

Tip 4 Assure that every low-stakes assessment provides meaningful feedback for learning

풍부한 정보량은 PA의 핵심이다. 의미있는 피드백이란 다양한 형태를 띌 수 있다.

Information richness is the cornerstone of programmatic assessment. Without rich assessment information programmatic assessment will fail. Mostly, conventional feedback from assessments, that is, grades and pass/fail decisions, are poor information carriers (Shute 2008). Meaningful feedback may have many forms. 

• 한가지는 시험이 종료된 이후에 정답과 오답에 대한 정보를 제공해주는 것이다.
  One is to give out the test material after test administration with information on the correct or incorrect responses. In standardised testing, score reports may be used that provide more detail on the performance (Harrison et al. 2013), for example, by giving online information on the blueprint categories of the assessment done, or on the skill domains (i.e. in an OSCE), or longitudinal overview for progress test results (Muijtjens et al. 2010). 
• 구두로 제공되는 피드백도 있을 수 있다. 비표준화된 평가에서 rating scale을 활용한 양적 정보를 얻기도 하지만, 한계가 있고 복잡한 기술에 대한 피드백은 묘사적 정보를 통하는 것이 더 낫다.
  Sometimes verbal feedback in or after the assessment may be given (Hodder et al. 1989). In unstandardised assessment, quantitative information usually stems from the rating scales being used. This is useful, but it also has its limitations. Feedback for complex skills is enhanced by narrative information (Govaerts et al. 2007). 
• 묘사적 정보는 표준화된 평가도 더 풍부하게 만들 수 있다.
  Narrative information may also enrich standardised assessment. For example, in one implementation of programmatic assessment narrative feedback is given to learners on weekly open-ended questions (Dannefer & Henson 2007). 
• 수량화하기 어려운 것을 억지로 수량화시킨다면 평가대상의 의의를 상실하게 될 수도 있다. 또한 점수만 따기 위한 행동이나 학점 인플레이션을 유발할 수도 있다.
  Given the fact that putting a metric on things that are difficult to quantify may actually trivialise what is being assessed. Metrics such as grades often lead to unwanted side effects like grade hunting and grade inflation. 
• 또한 평점은 의도치않게 피드백 과정을 '망칠' 수도 있다. 
  Grades may unintentionally “corrupt” the feedback process. Some argue we should replace scores with words (Govaerts & Van der Vleuten 2013), particularly in unstandardised situations where complex skills are being assessed such as in clinical workplaces. This is not a plea against scores. Scoring and metrics are fine particularly for standardised testing. This is a plea for a mindful use of metrics and words when they are appropriate to use in order to provide meaningful feedback.

효과적인 피드백을 얻기 위한 과정은 길고 힘들다. 자원을 절약하는 것에 관심을 가지는 것도 좋지만, 양질의 피드백을 제공하는데는 결국 시간과 노력이 필요하다. 두 가지를 명심해야 할 것이다.

Obtaining effective feedback from teachers, supervisors or peers can be a tedious process, because it is time and resource intensive. Considering resource-saving procedures is interesting (e.g. peer feedback or automatic online feedback systems), but ultimately providing good quality feedback will cost time and effort. Two issues should be kept in mind when thinking about the resources. 

• 평가와 학습은 서로 얽혀 있다. 즉 가르치는 시간과 평가하는 시간이 명확하게 구분되지 않는다.
  In programmatic assessment, assessment and learning are completely intertwined (assessment as learning), so the time for teaching and assessment becomes rather blurred. 
• 쓸모없는 피드백을 자주 하는것보다는 가끔이라도 양질의 피드백을 하는 것이 낫다.
  Second, more infrequent good feedback is better than frequent poor feedback. Feedback reception is highly dependent on the credibility of the feedback (Watling et al. 2012), so the “less-is-more” principle really applies to the process of feedback giving. High-quality feedback should be the prime purpose of any individual data point. If this fails within the implementation, programmatic assessment will fail.

학습자에게 멘토링을 제공하라

Tip 5 Provide mentoring to learners

피드백만으로는 부족할 수 있다. 피드백은 이상적으로는 성찰적 대화의 한 부분이어야 하며, 멘토링은 그러한 대화를 만들어나가는 효과적인 수단이다.

Feedback alone may not be sufficient for learners to be heeded well (Hattie & Timperley 2007). Research findings clearly indicate that feedback, reflection, and follow-up on feedback are essential for learning and expertise development (Ericsson 2004; Sargeant et al. 2009). Reflection for the mere sake of reflection is not well received by learners, but reflection as a basis for discussion is appreciated (Driessen et al. 2012). Feedback should ideally be part of a (reflective) dialogue, stimulating follow-up on feedback. Mentoring is an effective way to create such a dialogue and has been associated with good learning outcomes (Driessen & Overeem 2013).

PA에서 멘토링은 피드백 과정과 피드밸 활용을 지원하기 위한 목적을 갖는다. 멘토의 역할. 멘토의 역할은 학습자에서 최대치를 이끌어내는 것이다. 전통적 평가에서 최소 기준을 만족하는 것이 진급을 위해 충분한 것이었다면 PA에서는 개인의 수월성을 추구하는 것이 목적이며, 멘토는 이러한 수월성 달성을 위한 핵심인물이다. 

In programmatic assessment mentoring is used to support the feedback process and the feedback use. In a dialogue with an entrusted person, performance may be monitored, reflections shared and validated, remediation activities planned, and follow-up may be negotiated and monitored. This is the role of a mentor. The mentor is a regular staff member, preferably having some knowledge over the curriculum. Mentor and learner meet each other periodically. It is important that the mentor is able to create a safe and entrusted relationship. For that purpose the mentor should be protected in having a judgemental role in the decision-making process (Dannefer & Henson 2007). The mentor’s function is to get the best out of the learner. In conventional assessment programmes, adherence to minimum standards can suffice for promotion and graduation. In programmatic assessment individual excellence is the goal and the mentor is the key person to promote such excellence.

신뢰할 수 있는 의사결정을 내려라

Tip 6 Ensure trustworthy decision-making

풍부한 정보를 담고 있는 자료는 보통 양적, 질적 자료의 특성을 모두 가지고 있기 때문에, 이러한 정보를 종합해서 판단하는 것은 전문가적 판단력이 필요하다. '고부담'이라는 특성을 감안했을 때, 이러한 판단은 충분한 신뢰성을 갖추어야 하며 절차적 방법론이 이러한 신뢰성의 근거가 되어야 한다. 다음의 절차를 포함할 수 있다.
High-stakes decisions must be based on many data points of rich information, that is, resting on broad sampling across contexts, methods and assessors. Since this information rich material will be of both quantitative and qualitative nature, aggregation of information requires professional judgement. Given the high-stakes nature, such professional judgement must be credible or trustworthy. Procedural measures should be put in place that bring evidence to this trustworthiness. These procedural measures may include (Driessen et al. 2013):

• An appointment of an assessment panel or committee responsible for decision-making (pass–fail–distinction or promotion decisions) having access to all the information, for example, embedded in the e-portfolio. Size and expertise of the committee will matter for its trustworthiness.
• Prevention of conflicts of interest and ensuring independence of panel members from the learning process of individual learners.
• The use of narrative standards or milestones.
• The training of committee members on the interpretation of standards, for example, by using exceptional or unusual cases from the past for training purposes.
• The organisation of deliberation proportional to the clarity of information. Most learners will require very little time; very few will need considerable deliberation. A chair should prepare efficient sessions.
• The provision of justification for decisions with high impact, by providing a paper trail on committee deliberations and actions, that is, document very carefully.
• The provision of mentor and learner input. The mentor knows the learner best. To eliminate bias in judgement and to protect the relationship with the learner, the mentor should not be responsible for final pass–fail decisions. Smart mentor input compromises can be arranged. For example, a mentor may sign for the authenticity of the e-portfolio. Another example is that the mentor may write a recommendation to the committee that may be annotated by the learner.
• Provision of appeals procedures.

This list is not exhaustive, and it is helpful to think of any measure that would stand up in court, such as factors that provide due process in procedures and expertise of the professional judgement. These usually lead to robust decisions that have credibility and can be trusted.

중간 의사결정을 위한 평가를 조직하라

Tip 7 Organise intermediate decision-making assessments

모든 과정이 끝나고 이루어지는 '고부담 결정'은 학습자를 놀래키는 식으로 진행되어서는 안 된다. 중간평가의 결과를 제공하고 최종 결정에 대한 피드백을 줌으로써 최종평가의 신뢰성을 높일 수 있다. 중간평가는 보다 작은 수의 데이터포인트를 기반으로 내려진다. '저부담'과 '고부담'사이의 '중부담' 평가라 할 수 있으며, 진단적/치료적/예후적 역할을 할 수 있다. 이상적으로는 평가위원회가 모든 중간평가 결과를 제공하는 것이 좋으나 모든 학생에 대한 전체 위원회 평가를 하는 것은 지나치게 자원이 많이 소모될 것이다. 따라서 보다 자원을 효율적으로 사용할 수 있는 접근법을 고려할 필요가 있다.

High-stakes decisions at the end of the course, year, or programme should never be a surprise to the learner. Therefore, provision of intermediate assessments informing the learner and prior feedback on potential future decisions is in fact another procedural measure adding to the credibility of the final decision. Intermediate assessments are based on fewer data points than final decisions. Their stakes are in between low-stake and high-stake decisions. Intermediate assessments are diagnostic (how is the learner doing?), therapeutic (what should be done to improve further?), and prognostic (what might happen to the learner; if the current development continues to the point of the high-stake decision?). Ideally, an assessment committee provides all intermediate evaluations, but having a full committee assessing all students may well be a too resource-intensive process. Less costly compromises are to be considered, such as using subcommittees or only the chair of the committee to produce these evaluations, or having the full committee only looking at complex student cases and the mentors evaluating all other cases.

개별화된 교정교육을 장려하고 촉진하라.

Tip 8 Encourage and facilitate personalised remediation

교정교육은 재시험과는 다르다. 교정교육은 지속적인 성찰과정에서 드러나는 진단적 정보를 바탕으로 이뤄져야 하며, 언제나 개별화되어야 한다. 따라서 교육과정은 학습자가 교정교육을 계획하고 이수할 수 있도록 충분한 유연성이 있어야 한다. 비용이 많이 드는 교정교육 패키지를 개발할 필요는 없으며 학습자를 어떤 교정교육이 필요할지에 대한 결정에 참여시키고, 경험이 풍부한 멘토로부터 지원을 받도록 하면 된다. 이상적으로 교정교육은 충분한 지원과 방법을 학습자에게 제공하여 스스로의 책임이 되도록 해야 한다.

Remediation is essentially different from resits or supplemental examinations. Remediation is based on the diagnostic information emanating from the on-going reflective processes (i.e. from mentor meetings, from intermediate evaluations, and from the learner self) and is always personalised. Therefore, the curriculum must provide sufficient flexibility for the learner to plan and complete remediation. There is no need for developing (costly) remediation packages. Engage the learner in making decisions on what and how remediation should be carried out, supported by an experienced mentor. Ideally, remediation is made a responsibility of the learner who is provided with sufficient support and input to achieve this.

프로그램의 효과와 활용을 모니터하고 평가하라

Tip 9 Monitor and evaluate the learning effect of the programme and adapt

멘토는 중요한 이해관계자이다.

Just like a curriculum needs evaluation in a plan-do-act-cycle, so does an assessment programme. Assessment effects can be unexpected, side effects often occur, assessment activities, particularly very routine ones, often tend to trivialise and become irrelevant. Monitor, evaluate, and adapt the assessment programme systematically. All relevant stakeholders involved in the process of programmatic assessment provide a good source of information on the quality of the assessment programme. One very important stakeholder is the mentor. Through the mentor’s interaction with the learners, they will have an excellent view on the curriculum in action. This information could be systematically gathered and exchanged with other stakeholders responsible for the management of the curriculum and the assessment programme. Most schools will have a system for data-gathering on the quality of the educational programme. Mixed-method approaches combining quantitative and qualitative information are advised (Ruhe & Boudreau 2013). Similarly, learners should be able to experience the impact of the evaluations on actual changes in the programme (Frye & Hemmer 2012).

평가절차에서 나온 정보를 교육과정 평가에 활용하라

Tip 10 Use the assessment process information for curriculum evaluation

평가는 주로 세 가지 기능을 한다.

Assessment may serve three functions: 

• to promote learning, 
• to promote good decisions on whether learning outcomes are achieved, and 
• to evaluate the curriculum. 

In programmatic assessment, the information richness is a perfect basis also for curriculum evaluation. The assessment data gathered, for example, in the e-portfolio, not only provides an X-ray of the competence development of the learners, but also on the quality of the learning environment.

이해관계자간 지속적 상호작용을 하라

Tip 11 Promote continuous interaction between the stakeholders

PA는 모든 사람들에게 영향을 미친다. 따라서 교육기관 전체에 대한 책임이 있다. 의사소통이 중요하며, 의사소통은 불완전성을 의미할 수도 있다. 평가위원과 멘토 사이에 벽이 있다면 객관적이고 독립적인 의사결정이 가능할지는 모르겠지만, 정보는 그만큼 덜 풍부해지는 것이다. 

As should be clear from the previous, programmatic assessment impacts at all levels: students, examiners, mentors, examination committees, assessment developers, and curriculum designers. Programmatic assessment is, therefore, the responsibility of the whole educational organisation. When implemented, frequent and on-going communication between the different stakeholder groups is essential in the process. Communication may regard imperfections in the operationalisation of standards or milestones, incidents, and interesting cases that could have consequences for improvement of the system. Such communication could eventually affect procedures and regulations and may support the calibration of future decisions. For example, a firewall between the assessment committee and mentors fosters objectivity and independency of the decision-making, but at the same time may also hamper information richness. Sometimes, however, decisions need more information about the learner and then continuous communication processes are indispensable. The information richness in programmatic assessment enables us to make the system as fair as possible.

도입을 위한 전략을 개발하라

Tip 12 Develop a strategy for implementation

PA는 학습에 대한 구성주의적 관점을 기반으로 한다. 평가시스템을 급격하게 변화시키는 것은 평가가 무뎌지거나 학생들의 'gaming'에 취약해질 것이라는 우려를 가지게 하나, 실제 활용한 사례를 살펴보면 그 반대이다. 그럼에도 고등교육의 많은 부분이 변화에 저항하는 특성이 있어서 변화전략이 필요하다. 

Programmatic assessment requires a culture change in thinking about assessment that is not easy to achieve in an existing educational practice. Traditional assessment is typically modular, with summative decisions and grades at the end of modules. When passed, the module is completed. When failed, repetition through resits or through repetition of the module is usually the remedy. This is all very appropriate in a mastery learning view on learning. However, modern education builds on constructivist learning theories, starting from notions that learners create their own knowledge and skills, in horizontally and/or vertically integrated programmes to guide and support competence. Programmatic assessment is better aligned to notions of constructivist learning and longitudinal competence development through its emphasis on feedback, use of feedback to optimise individual learning and remediation tailored to the needs of the individual student. This radical change often leads to fear that such assessment systems will be soft and vulnerable to gaming of students, whereas the implementation examples demonstrate the opposite effect (Bok et al. 2013). Nevertheless, for this culture change in assessment a change strategy is required, since many factors in higher education are resistant to change (Stephens & Graham 2010). A change strategy needs to be made at the macro-, meso- and micro levels.

• At the macro level, national legal regulations and university regulations are often strict about assessment policies. Some universities prescribe grade systems to be standardised across all training programmes. These macro level limitations are not easy to influence, but it is important to know the “wriggle room” these policies leave for the desired change in a particular setting. Policy-makers and administrators need to become aware of why a different view on assessment is needed. They also need to be convinced on the robustness of the decision-making in an assessment programme. The qualitative ontology underlying the decision-making process in programmatic assessment is a challenging one in a positivist medical environment. Very important is to explain programmatic assessment in a language that is not jargonistic and which aligns with the stakeholder’s professional language. For clinicians, for example, analogies with diagnostic procedures in clinical health care often prove helpful.
• At the meso level programmatic assessment may have consequences for the curriculum. Not only should the assessment be aligned with the overarching competency framework, but with the curriculum as well. Essential are the longitudinal lines in the curriculum requiring a careful balance of modular and longitudinal elements. Individual stakeholders and committees need to be involved as early as possible. Examination rules and regulations need to be constructed which are optimally transparent, defensible, but which respect the aggregated decision-making in programmatic assessment. The curriculum also needs to allow sufficient flexibility for remediation. Leaders of the innovation need to be appointed, who have credibility and authority.
• Finally, at the micro level teachers and learners need to be involved in the change right from the start. Buy-in from teachers and learners is essential. To create buy-in the people involved should understand the nature of the change, but more importantly they should be allowed to see how the change also addresses their own concerns with the current system. Typically, teaching staff do have the feeling that something in the current assessment system is not right, or at least suboptimal, but they do not automatically make the connection with programmatic assessment as a way to solve these problems.

The development of programmatic assessment is a learning exercise for all and it is helpful to be frank about unexpected problems to arise during the first phases of the implementation; that is innate to innovation. So it is therefore good to structure this learning exercise as a collective effort, which may exceed traditional faculty development (De Rijdt et al. 2013). Although conventional faculty development is needed, involving staff and students in the whole design process supports the chance of success and the creation of ownership (Könings et al. 2005) and creates a community of practice promoting sustainable change (Steinert 2014).

PA로 변화하는 것은 전통적 교육과정이 PBL로 변화하는 과정에 비견될 수 있다. 

Changing towards programmatic assessment can be compared with changing traditional programmes to problem-based learning (PBL). Many PBL implementations have failed due to problems in the implementation (Dolmans et al. 2005). When changing to programmatic assessment, careful attention should be paid to implementation and the management of change at all strategic levels.

Conclusion

Programmatic assessment has a clear logic and is based on many assessment insights that have been shaped trough research and educational practice. Logic and feasibility, however, are inversely related in programmatic assessment. To introduce full-blown programmatic assessment in actual practice all stakeholders need to be convinced. This is not an easy task. Just like in PBL, partial implementations are possible with programmatic assessment (i.e. the increase in feedback and information in an assessment programme, mentoring). Just like in PBL, this will lead to partial success. We hope these tips will allow you to get as far as you can get.

12 Tips for programmatic assessment.

MMI로 평가한 인적특성의 변동요인(Medical Teacher, 2014)

Variance in attributes assessed by the multiple mini-interview

NIKKI BIBLER ZAIDI1, CHRISTOPHER SWOBODA2, LEIGH LIHSHING WANG2 & R. STEPHEN MANUEL3

1University of Michigan Medical School, USA, 2University of Cincinnati, USA, 3University of Cincinnati College of Medicine, USA

Introduction

가장 신뢰도가 높은 입학면접은 어떤 형태일 것인가?, MMI에 대한 논의. MMI의 다면표집법은 낮은 신뢰도를 극복하는 수단으로 긍정적 평가를 받았다.

The medical school preadmission interview (MSPI) remains a widely used tool in medical school admissions (Monroe et al. 2013); therefore, discussions regarding the most reliable and valid MSPI format continue to evolve (Edwards et al. 1990; Goho & Blackman 2006). Over the past decade, the multiple mini-interview (MMI) has gained considerable attention as an alternative to more traditional MSPI formats. The MMI is a multi-sampling, structured interview format in which interviewers, referred to as “raters,” assess specific applicant attribute(s) using multiple 5–15 minute interview stations. Each interview station is assigned a different discussion prompt, referred to as a “scenario;” likewise, each station has a different rater who is tasked with assigning applicants scores for a set of items on an evaluation tool (Eva et al. 2004c; Pau et al. 2013). The MMI’s multi-sampling technique has been celebrated for increasing the low reliability estimates that plague traditional MSPIs (Eva et al. 2004b, c; Uijtdehaage et al. 2011), and some studies suggest that MMI scores can be used to predict performance during medical school clerkships and on medical licensure examinations (Eva et al. 2004a, 2009, 2012).

MMI의 신뢰도는 흔히 G theory를 이용해서 추정된다. 어떤 측정이든 (MMI를 포함하여) 그 목적은 진점수(true score)를 흐릿하게 만드는 원하지 않는 변이(unwanted variance)를 줄이는 것이다. G theory를 활용한 대부분의 MMI에 대한 연구에서 그 모델은 평가자와 스테이션을 facet으로 모델링하였다.

Reliability of the MMI is commonly estimated using Generalizability (G) theory because of the multi-faceted nature of the measurements. In any measurement process, including the MMI, the goal is to reduce the unwanted variance in observed scores that can obscure true scores. G theory can simultaneously capture multiple sources of unwanted variance, referred to as “facets,” to provide an estimate of generalizability – or reliability (Brennan 2001). Consequently, MMI reliability is expressed as a G coefficient and represents a “universe of admissible observations” – a “universe” that is defined by the specific facet(s) that the researcher decides to include in the model. The decision regarding facets for inclusion is based on the context of a measurement to which the researcher plans to generalize findings. For instance, if some raters are always more lenient or more severe than other raters, then raters are a source of unwanted variance in MMI scores, and the rater facet would be modeled in a subsequent G study if the researcher wishes to generalize across raters. Most MMI studies associated with medical school admissions have modeled raters and/or stations as facets (Eva et al. 2004b, c; Uijtdehaage et al. 2011).

각 facet은 condition으로 구성되는데, 조건(condition)은 CTT에서 factor의 수준에 해당하는 것이다. 연구자들은 Condition에 대해서, condition은 어떤 측정의 질을 낮추지 않으면서도 바꿀 수 있다라고 가정한다. 

Each facet that defines the universe of admissible observations is comprised of “conditions” (Brennan 2001). These conditions are analogous to the levels of a factor in classical test theory (CTT). Overall, the MMI literature reports facets with a range of corresponding conditions and it is presumed, as an assumption for most applications of G theory, that the varying conditions represent random samples from these facets. Therefore, it is also generally assumed by researchers that these conditions can be altered without making the measurement any less acceptable. Although G theory can examine the extent to which such changes in a facet’s conditions make the measurement more or less acceptable (Shavelson & Webb 1991), this concept of interchangeability has not been examined for all potential facets of the MMI.

MMI에 대한 G coefficient연구에서 rater와 station을 facet으로 했지만, 어떤 인적특성을 평가하는가는 대체로 무시해왔던 것이 사실이다. MMI에서 평가하는 인적특성은 지금까지는 잘 포함시키지 않아왔지만 상당히 큰 변이(variance)의 원인이 될 수 잇다. 여러 연구에서 의사에게 요구되는 서로 다른 인적특성을 최대 87개까지 추출한 바 있지만, 입학면접에서는 그 중 일부만을 평가할 수 있을 뿐이다. 따라서 MMI에서 평가되는 구체적인 인적특성은 의과대학마다 리더십, 문화적 감수성, 대인관계, 비판적 사고 등으로 다양할 것이다. 또한 이들 평가는 주로 Likert scale로 평가하게 되며, 평가대상이 되는 인적특성은 의사로서 중요한 다양한 특성 중 무작위로 선정된다는 암묵적 가정을 기반으로 한다. 즉, MMI 연우게서는 이러한 인적특성들이 item 측면에서 상호교환가능함을 가정하고 있다. 즉, '리더십'에 대한 점수는 '문화적 감수성'에 대한 점수와 동등하고 상호교환가능하다는 뜻이다. 연구자들은 이러한 가정의 안면타당도에 의문을 제기하였으며 추가적 연구가 필요하게 되었다.

The extant literature reports moderate to high G coefficients for medical school MMIs ranging 0.58–0.81 (Eva et al. 2004b, c; Uijtdehaage et al. 2011). These reports, however, are based on studies that have modeled raters and stations as facets but have essentially ignored the impact of the attributes assessed. The attributes assessed by the MMI have the potential to introduce additional and largely unaccounted for, variance in MMI scores. The medical literature identifies up to 87 different attributes considered important for an aspiring physician (Price et al. 1971; Albanese et al. 2003); yet, an MSPI, including the MMI, can only reasonably capture a handful of these attributes. Therefore, the specific attributes assessed by an MMI will vary across medical schools and can range from leadership potential, cultural sensitivity, interpersonal skills, and critical thinking to a single, overall performance score (Eva et al. 2004c; Reiter et al. 2007; Uijtdehaage et al. 2011). These attributes are generally assessed as items on a Likert-like scale (Eva et al. 2004c) and carry the implicit assumption that an institution’s choice of attribute(s) can be considered a random selection from the domain of characteristics deemed important for the medical profession. Subsequently, MMI studies have largely considered attributes to be interchangeable conditions within the item facet. This would suggest that it is reasonable to believe that scores for the item, “leadership potential,” are parallel and interchangeable with scores for the item, “cultural sensitivity.” Consequently, the researchers question the face validity of this assumption and believe it warrants further investigation.

더 나아가서 MMI에서 평가하는 인적특성의 구성은 item facet을 넘어서 station facet으로 들어간다. 각 MMI 스테이션은 특정한 주제에 맞는 특정한 시나리오를 가지고 진행되는데, 이 시나리오를 바탕으로 평가서식에 의해서 'item'화 되는 사전에 결정된 인적특성에 대해 평가하게 된다. 결과적으로 station scenario 사이의 차이는 한번 더 의도하지 않은 변이를 유발할 수 있다. 기존 연구들은 station facet을 포함시키기는 했으나, station facet을 1회의 측정사건(measurement occasion)에만 국한시켰다. 따라서 기존 문헌에서 측정사건의 숫자가 증가할수록 MMI의 일반화가능도가 높아지는 것으로 되어있으므로 기존 연구의 결과는 CTT의 Spearman-Brown prophecy formula에만 부합하는 것일 수 있다. 

Furthermore, the impact of the composition of attributes assessed by the MMI has the potential to reach beyond the item facet into the station facet. Each MMI station is assigned a specific scenario that focuses on topics such as “knowledge of the healthcare system” or “critical thinking” and is intended to elicit information regarding a set of predetermined attributes that are captured as items on an evaluation form (Eva et al. 2004c). Consequently, differences among station scenarios have the potential to introduce further unwanted variance in the attributes assessed. Previous studies model the station facet into G studies (Eva et al. 2004b, c; Uijtdehaage et al. 2011); however, these studies generally recognize the station facet in terms of a measurement occasion only. Therefore, while it is well-established in the literature that increasing the number of measurement occasions increases generalizability estimates for the MMI, this finding merely aligns with the CTT’s Spearman-Brown prophecy formula. 

결과적으로 기존의 문헌은 스테이션의 시나리오가 MMI에서 인적특성을 평가하는데 미치는 영향력을 상당부분 무시해왔다고 볼 수 있다. MMI가 맥락특이성을 희석시키기 위한 목적으로 개발되었다는 점을 고려하면, 이에 대한 추가적 연구가 필요하다. 실제로 Eva의 파일럿연구를 보면, 지원자-스테이션 상호작용이 지원자 단독으로 인한 변이보다 다섯배나 컸다. 이러한 결과는 스테이션의 내용이 MMI점수에서 발생하는 오차의 중요한 원인이 될 수 있음을 보여준다. 그러나 아직까지 어떤 연구도 특정 인적특성(즉 item)에 대해서 MMI 스테이션의 내용, 즉 시나리오가 의과대학 지원자의 평가에 어떤 영향을 주는가를 연구한 바는 없다. 본 연구에서는 MMI평가서식의 구체적 특성에 의해서 정의내려진 item이 시나리오에 관계없이 여러 MMII station에 걸쳐서 일관되게 평가되어지는지를 연구해보고자 한다.

Consequently, the extant literature has largely ignored the potential influence of the stations’ scenario on the assessment of attributes within an MMI. Given the fact that the MMI was created in large part to dilute the effects of context specificity (Eva 2003), this warrants further investigation. In fact, Eva et al.’s (2004c) pilot study concluded that variance attributable to the candidate–station interaction was five times greater than that assigned to the candidate alone. This finding suggests that station content may introduce the most significant source of error in MMI scores. Yet, to the best of the authors’ knowledge, no study has examined how the MMI station content – the scenario – may influence the assessment and evaluation of medical school applicants on a set predetermined attributes (i.e. items). This study will explore whether items, defined as specific attributes on an MMI evaluation form, are assessed consistently across MMI stations regardless of station scenario.

Methods

This study examines one aspect of psychometric evidence from one United States (US) medical school that has fully adopted the MMI process as a replacement for the traditional MSPI. Using G theory, this study examines the variance attributable to the item facet and the scenario-item interaction. Data used for analysis represent MMI scores that were collected for the sole purpose of making admissions decisions. These data come from a US medical school that receives approximately 4000 admissions applications annually and interviews approximately 625 applicants each year. This institution fully adopted the MMI to select the entering class of 2009. With IRB approval (# 10-06-08-01), all applicants who participated in the MMI from 2009 to 2013 are included in the dataset used for analysis. This empirically collected dataset represents a nested design; therefore, only a small subset of applicants was used in this analysis in order to create a fully crossed design because in G theory, nested facets make it impossible to estimate all variance components separately (Brennan 2001).

'의사소통'과 이를 평가하기 위한 여섯 개의 구체적 특성 + 하나의 총괄평가

After a comprehensive blueprinting process, the school’s Admissions Committee identified one overarching characteristic – communication – to assess through the MMI. The rationale for choosing this single construct was largely rooted in literature that suggests that one of the chief patient complaints concerns poor communication between the patient and physician (Wofford et al. 2004). Communication was selected as the single construct from the larger domain of attributes deemed important for an aspiring physician. To operationalize this construct, six specific attributes and one “overall score” were used as sub score items in the MMI evaluation tool. The specific attributes assessed by this MMI included (1)multiple perspectives, (2)reflection of scenario, (3)articulation, (4)interest in dilemma, (5)non-verbal communication, and (6)interpersonal skills. These seven items were measured on a seven-point Likert-like scale that assumes equal intervals between the anchors (Unsatisfactory-1; Below Average-2; Slightly Below Average-3; Average-4; Slightly Above Average-5; Above Average-6; Outstanding-7).

G-String IV software (Bloch & Norman, Hamilton, Ontario, Canada), was used to estimate variance components attributable to the facets of measurement. In G theory, the object of measurement is not considered a facet. Therefore, this study’s two-facet design includes the object of measurement – applicants (p) – and two facets of generalization – scenario (s) and item (i).

Facet of differentiation

The object of measurement is considered the facet of differentiation. This facet of differentiation is analogous to the dependent variable and is considered the only desired source of variation. In other words, the object of measurement is the “universe” or “true” score (Brennan 2001). The facet of differentiation is the person, the applicant (p) facet, which represents the true MMI score for the applicant. Therefore, this variance should be large and other modeled sources of variance are expected to be small.

Facets of generalization

The facets of generalization are analogous to the independent variables and they contribute unwanted sources of error to the universe score, or for this study – MMI scores. These facets of generalization include the sources of measurement error that the researcher intends to generalize from the sample to the universe of admissible observations. Because this study intends to generalize applicant scores from one scenario to applicant scores from a much larger set of scenarios, scenario (s) is considered a facet of generalization. Likewise, because this study intends to generalize from applicant scores on one attribute item to applicant scores on a much larger set of items, item (i) is also a facet of generalization. In line with G theory assumptions, both the scenario facet and the item facet are considered random and conditions within these facets are deemed interchangeable (Shavelson & Webb 1991).

Confounded facet

Because there is one rater assigned to each scenario at this US medical school, the variance attributable to rater cannot be disentangled from variance attributable to scenario. Therefore, rater and scenario variance are completely confounded. For the purposes of this study; however, this confounded effect will be recognized as a limitation and the variance accounted for by this confounded facet will be considered attributable to the scenario (s).

Sample

지원자에 대한 MMI점수가 서로 다른 시나리오 아래서 수집되는 nested structure이다. 따라서 scenario facet이 applicant facet과 fully crossed 되는 subset을 찾는 purposive sampling을 하였음. 결과적으로 completely crossed design을 위하여 동일한 시나리오에서 동일한 아이템으로 평가받은 지원자를 표집하였다.

This study uses actual admissions data; therefore, the data structure represents a pragmatic design in which inevitable nesting and confounding occurs. A fully crossed G study elicits the most information; however, the existing data set represents a nested structure in which MMI scores are collected for applicants by using different scenarios. Therefore, a purposive sampling method was employed to generate a subset of data in which the scenario (s) facet (confounded but representing the same raters within scenario combinations) was fully crossed with the applicant (p) facet. Consequently, a subset of the full dataset was intentionally sampled for applicants rated within the same scenario using the same items to ensure a completely crossed design. The sample included 16 applicants who were evaluated within the same six scenarios and scored on the same seven items. This small, purposive sample was necessary in order to examine the variance attributable to the main effect of the scenario (s) facet and the scenario- item (si) interaction (Shavelson & Webb 1991), which is information pertinent to the study’s objectives.

Results

While the true score (p) should represent a sizable amount of variance, Table 1 shows that the applicant (p) represents only 6% of total variance. The estimated variance components from the G study suggest that the greatest amount of variance is attributable to the main effect of the scenario (s) facet and the interaction between scenario and applicant (ps). Collectively, these two variance components account for 77% of the total variance. The item facet (i) represents the lowest estimated variance, estimating only 0.6% of the total variance in MMI scores. Likewise, the scenario-item interaction (si) accounts for only 1.4% of the total variance. The low estimate of variance attributable to the item facet is reinforced by a high Cronbach’s alpha (0.97) for the seven items, which suggests very high internal consistency among the attributes measured by this MMI.

Discussion

일곱 개의 sub scores (items)의 높은 내적 일관성으로부터 현 MMI에서는 하나의 단일한 차원의 인적특성을 평가하고 있음을 알 수 있다. item facet으로부터 유발되는 variance가 2%에 불과한 것도 이를 지지한다. 만약 이 item들이 하나의 단일차원의 특성을 평가하는 것이라면 일곱 개의 item은 하나의 item으로 압축될 수 있다. p와 i에 의한 변이가 적다는 점은 대부분의 변이가 s에 기인한다는 것을 의미한다.

The high internal consistency of the seven sub scores (items) may support assumptions that the current MMI process is measuring one unidimensional attribute; this is further supported by only 2% of variance attributable to the item facet– (i), (pi), and (si). These findings either support the G theory assumption that conditions of the item facet can be considered interchangeable or it may suggest that raters do not understand how to use the items associated with the MMI evaluation tool and simply assign the same value for each item. If the items are capturing one unidimensional attribute, then a seven-item evaluation tool could be condensed into a single item tool. The low percentage of variance attributable to both items (i) and the true score – the applicant (p) – further suggests that the variation in MMI scores is mostly attributable to scenarios (s). 

스테이션 시나리오의 내용에 차이가 있다는 점을 감안하면, 시나리오에 의해서 지원자가 보여주는 인적특성의 비일관성이 높아진다고 보는 것이 타당하다. 예컨대 지원자는 윤리적 딜레마를 포함하고 있는 시나리오와 팀워크 활동을 포함하는 시나리오에서 서로 다른 특질을 보여줄 것이다. 따라서 MMI가 item차원에서는 하나의 단일차원 특성(one unidimensional attribute)을 측정하게끔 한다 하더라도, 스테이션의 내용은 그 특성(attribute)에 대한 측정을 변화시킴으로서 시나리오 수준에서의 다차원(multidimensionality)을 유발 할 수 있다. 연구자들은 이러한 차이가 시나리오-아이템 상호작용으로부터 나타날 것으로 기대했으나 본 연구의 결과는 이러한 가설이 틀렸음을 보여준다. 아마도 item facet으로 인한 변이의 비율이 낮기 때문에 이런 결과가 나왔을 것이다. 따라서 시나리오-아이템 상호작용이 작다는 것은 item facet으로 인한 variance가 scenario facet에 의한 variance에 포함되어버리기 때문일 수 있다. 결과적으로 이러한 상호작용이 MMI점수의 variance중 상당한 부분을 차지하게 될 것이나, 이러한 것이 이번 연구 샘플에서는 드러나지 않았다. 

Given the variation among the content of station scenarios, it is plausible to believe that scenarios promote inconsistencies among attributes exhibited by an applicant. For instance, a scenario involving an ethical dilemma might highlight different attributes than a scenario requiring an applicant to engage in a teamwork activity. Therefore, even if the MMI is supposedly measuring one unidimensional attribute at the item level, the content of the stations may elicit different measurements of attributes, thereby introducing multidimensionality at the scenario level. While the researchers expected to find this disparity manifested as a large variance component associated with the scenario-item interaction, this initial analysis does not support the assumption. This potential effect may be obscured by the low percentage of variance attributable to the item facet. Therefore, it is possible that the small scenario-item interaction is a result of variance attributable to the item facet being subsumed by the variance attributable to the varying conditions of the scenario facet. Consequently, the interaction may indeed contribute substantial variance in MMI scores; but this was not identified within this study’s sample.

AERA, APA, NCME 기준에서 드러나듯, "만약 문항 개발자가 시험을 수행하는 조건이 응시자에 따라서 다를 수 있음을 적시한다면, 그러한 조건에서 허용가능한 변이가 확인되어야 하고, 서로 다른 조건을 인정하는 rationale가 명시되어야 한다"라고 언급하고 잇다. item facet으로부터 기인하는 변이가 작다는 것이 item facet이라는 서로 다른 조건에 대한(즉 서로 다른 특성들에 대한) 허용가능성을 의미할 수 있지만, 이러한 가정은 scenario facet에는 해당되지 않는다. 본 연구의 결과는 scenario facet이라는 조건에 대한 상호교환가능성에 의문을 제기한다. 따라서 시나리오 선정에 보다 주의를 기울일 필요가 있다.

As outlined by the AERA, APA and NCME Standard 3.21, “If the test developer indicates that the conditions of administration are permitted to vary from one test taker or group to another, permissible variation in conditions for administration should be identified, and a rationale for permitting the different conditions should be documented” (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education 1999, p. 47). While the low variance attributable to the item facet may suggest permissibility for permitting different conditions of the item facet (i.e. attributes), this assumption may not hold for the conditions of the scenario facet. The results of this study suggest that interchangeability for the conditions of the scenario facet is questionable. Subsequently, more attention should be directed towards the selection of scenarios.

이러한 연구결과에도 불구하여 몇 가지 한계가 있다.

Despite these findings, this study has some limitations. Because this study uses empirically collected data intended for admissions purposes, the researchers did not have direct control over the data collection process. Consequently, this G study is limited by the need for a purposive sample which results in a very small sample size relative to the larger dataset. The researchers felt that the benefit of using a fully crossed design justified the small sample size. In G theory, the nested facet cannot be estimated separately from its interaction effects because nesting creates missing cells in the design; additionally, the scenario-item (si) interaction, a major focus of this study, could only be examined using a fully crossed design (Brennan 2001). Nonetheless, the sample size used in this study may not be representative of the larger population; subsequently, the variance components might be influenced by sampling error. Because estimated variance components can be very unstable when the number of conditions within a measurement is small, this study could be replicated using a larger sample size. In addition, only two facets of generalization are modeled and one of these facets, the scenario facet, is confounded with another potential source of variation – the rater facet. Therefore, other facets could be added to the model in order to expand the universe of admissible observations and the corresponding generalizability of the study. Consequently, this study’s external validity is limited to the extent that other MMIs mirror the one used in this analysis. Despite these limitations, this study offers a solid framework for future exploration into the impact that scenario content can have on the attributes assessed by the MMI.

Conclusions

Because of the variation in how and what an institution-specific MMI measures, psychometric properties must be examined for each medical school that chooses to adopt the MMI as a replacement for the MSPI. This study adds to the growing body of literature related to psychometric analyses of the MMI. Because the extant literature has primarily focused on predictive validity and largely ignored other aspects of validity, this study adds to the foundation for further exploration into construct validity. As the MMI continues to gain momentum as a replacement for the traditional MSPI, the measurement process deserves careful attention, especially in terms of how and what is measured. Future analysis should explore the potential that both items and scenarios have on subsequent MMI scores. Overall, the results of this study reinforce the need to examine all psychometric properties of a measurement process – especially one, such as the MMI, that is used for high-stakes admissions purposes.

Variance in attributes assessed by the multiple mini-interview.

교육과정 개편을 위한 열두가지 팁(Medical Teacher, 2015)

Twelve tips for curriculum renewal

PETER MCLEOD & YVONNE STEINERT

McGill University, Canada

서론 Introduction

교육과정의 정의. 교육과정은 변화에 끊임없이 반응해야 하며, 교육리더, 교수, 학습자는 근거를 기반으로 한 정기적 개선의 필요성을 인정하고 그 가치를 존중해야 한다.

A curriculum is “a planned educational experience that encompasses behavioral goals, instructional methods and actual experiences of the learners” (Green 2001). When a new curriculum has been developed and successfully launched, a forward looking process of curriculum renewal should be a major priority. Renewal should be characterized by thoughtful evaluation, revision, ongoing responsiveness and modernization. To assure that the curriculum remains responsive to emerging societal trends, health care innovations, and novel education practice, the renewal should be a dynamic process designed to enhance student learning. All health sciences curricula should be modified and regenerated on a regular basis (Davis & Harden 2003). Given the ever-changing circumstances influencing the conduct of learning, curriculum developers and reviewers must embrace flexibility and ongoing responsiveness to meet teacher, student and societal expectations. In other words, the curriculum should be perpetually responsive to change and regular evidence-based renewal must be accepted and valued by educational leaders, teachers and learners.

교육과정을 바꿔야 하는, 그리고 지속적으로 개선해야 하는 근거를 분명히 하라

Tip 1 Articulate the reasons for curricular change and ongoing curricular renewal

변화의 이유는 무한하다.

The forces for change in health sciences education are legion and unremitting. Among the prominent forces are: societal pressures; shifting disease patterns as exemplified by the obesity epidemic and the prevalence of Type 2 diabetes; and new understandings of educational theory and practice. Proliferation of novel, expensive technologies, including sophisticated simulation materials, also have a role in influencing curricular change.

끊임없는 위험 중 하나는 '정보의 과다'이다. 1899년 Olser는 '학생은 너무 많이 배우려고 하고, 우리 교수들은 너무 많이 가르치려고 하는데, 둘 다 그다지 성공적이지는 못했다'라고 했다. 약 100년 후 Newman은 '학생들의 기억력에 가해지는 부담을 경감해줄 어떤 수단이 필요하다'라고 했다.

One unremitting risk in health science curricula is “information overload”. The information overload problem, by itself, is reason enough for change in health sciences curricula. History reminds us of the persistence of the “overload” problem. In November, 1899 William Osler wrote in the Montreal Medical Journal that The students try to learn too much and we teachers try to teach them too much, neither perhaps with much success. Almost 100 years later GB Newman wrote, The load upon students’ memories has become excessive as to require some measure of relief (Newman 1988).

(...)

Regardless of the reason for change or renewal, be it a recurring problem or a new trend, the motivation must be clearly stated, relevant and meaningful. Many change efforts, including those designed to develop curricula, fail in a predictable fashion. An easily predictable circumstance for failure is for the curriculum development committee members to deliberate wisely, celebrate the completion of the process, and then disperse.

Kotter가 제시한 프레임워크. 총 8단계 중에서 처음 두 단계가 Tip 1에 해당한다. 

Kotter provides a useful framework for leading change and the members of the development committee should familiarize themselves with his guidelines for leading change that are equally applicable to successful transformation of a curriculum (Kotter 2007). 

The first step in introducing change is to: “establish a sense of urgency” for change among the faculty. Failure to establish a sense of urgency may account for some of the failures to drive momentum-resistant people out of their comfort zones. An influential committee, or even a powerful duo, depending on the breadth of change contemplated, should be charged with “selling” the change and bringing other respected leaders on board. An influential committed coalition can help to inculcate other leaders into the change vortex by developing a picture of what they see as a vision for the future. 

Step 2, “communicating the vision”, is critical. This group of strong opinion leaders should be part of the vision process with a mandate to facilitate the next 6 steps of change as advocated by Kotter. They should be empowered to concentrate on efforts to spread the word about the curriculum change and to alert other school leaders that change will be followed by timely ongoing positive curriculum renewal. Real change takes time. As a result, the members of the development committee should recognize and celebrate achievement of short-term gains. Not all obstacles will easily fall by the wayside. Early successes will be appreciated by the committee members and the faculty members, including some who were skeptical of the need for change.

강력하고 영향력이 큰 교육과정 개편 팀을 구성하라

Tip 2 Build a strong, influential curriculum renewal team

다음의 사람들이 포함되어야 함.

To avert faculty ennui and to encourage “buy in”, it is advisable to develop a strong curriculum renewal team long before the curricular development committee has completed its job. Membership of the renewal team should include opinion leaders, course directors, high profile teachers and at least one member of the current curriculum committee. The size and makeup of the renewal team should be partly determined by the breadth and depth of anticipated curriculum renewal. Core team members should include faculty leaders in education and the health sciences as well as experienced teachers. The complexity and degree of curricular change envisioned by the development committee will also influence characteristics of the renewal group. For example, a curriculum for a brief learning experience, such as the introduction of students to a high-tech learning facility, can be developed by a small group of two or three persons from the renewal committee.

다음과 같은 업무를 해야 한다.

Ideally the curricular renewal team should be mandated to monitor the literature on, and respond to, influential curriculum research, disease prevalence, and technological innovations. There may be benefit to inclusion of competent, committed lay members on the renewal team to represent the changing needs of society, as well as practitioners from allied disciplines and schools. The renewal team members should canvas recent graduates of the course or program for their insights based on their experiences. Finally, faculty members with credible education expertise and understanding of education theory can be co-opted by the renewal committee during discussions of research-informed pedagogical practices.

유연성을 갖추어야 한다. "교육과정을 프레이밍하는 정답은 전체에 대한 것이든 부분에 대한 것이든 유일하지 않다. 교육과정은 목적에 부합하면 되며, 그 상황에 맞으면 된다."

A broadly representative renewal team as outlined above can mitigate any tendencies of committee members to filibuster for pet ideas. Flexibility must permeate the team discussions with the sense that “there is no one best choice for framing a curriculum as a whole or any of its parts. A curriculum should simply be fit for the purpose and context of its day” (Grant 2010).

사회, 정치, 경제, 기술적 트렌드를 파악하라

Tip 3 Identify prevalent social, political, economic and technological trends

전공의 세분화

At the outset of deliberations, the renewal team members should consider the influence of the increasingly narrow specialization of clinical departments and the limited representation of generalists in many health sciences teaching environments, including the environment at university-affiliated hospitals. Specialist practice profiles and “space age” technology can be irresistibly appealing for junior learners, so early student exposure to specialists may bias, and interfere with, the desired evolution of learning. Rare, “exciting” or esoteric disease is not the best field for learning. Learning from, and exposure to, common medical problems should be the norm.

상황학습

Situated learning, a process wherein learners transform because of participation in authentic activities in the clinic, office, or hospital ward, is most effective when there is deliberate exposure to generalists and to prevalent disease in the community (Weatherhall 2011). Worldwide, a number of troublesome influences have crept into health science education frameworks. Working conditions for health care providers who do the teaching have been threatened by daunting workloads and politically driven health reforms. Health care institutions which have strong profit-driven motives may not be ideal sites for learning. Even excellent community practices are at risk of being overstretched by increased patient expectations, thus limiting the opportunities for reflective teaching and learning.

국제적 질병부담

Changes in the local and global burden of disease have had worrisome influences on curriculum renewal as evidenced by the impact on students’ learning experiences of the AIDS epidemic, rising inner city violence, the obesity epidemic and the increasing prevalence of type 2 diabetes mellitus. Awareness of these issues must be considered in the curricular renewal process. On the other hand, skewed clinical experiences as a result of repeated exposure to dramatic, “interesting” medical problems may hamper leaning about common, important diseases.

만성질환자

Another challenge to learning in some active teaching hospitals is the overwhelming time commitment required of chronically ill patients whose care could be much better delivered in a long-term care facility. Effective curriculum renewal teams should insist that learner experiences be characterized by broad exposure to: excellent generalist role models; common important illnesses; and an environment wherein profit is not a primary driver for clinician teachers.

교육성과로부터 교수법과 학습법이 도출되도록 하라

Tip 4 Ensure that curriculum outcomes drive teaching and learning

용어를 둘러싼 혼란이 있다. 교육과정 목표를 개발하기 위하여, 그리고 학생과 교수들이 그것을 이해하게 하기 위한 수많은 작업들이 결국 실질적으로 적용하기에는 너무 부담스러운 긴 문서만을 남기는 경우가 허다하다.

At the outset of a renewal process it is useful to address the predictable, contentious issues related to important, but troublesome, education concepts and terms. “Objectives”, “Goals”, “Learning outcomes” and “Behavioral objectives” are but a few of the terms used to describe how and what students should learn in a curriculum. Confusion surrounding these terms invariably clouds discussions between teachers and curriculum framers. For example, development of useful behavioral objectives may be confounded by the difficulty of developing practical definitions (Rees 2004). Exercises designed to develop curriculum objectives, and to assure that learners and teachers understand them, frequently produce lengthy documents which are too cumbersome to be practical.

많은 교수들이 총괄적인 목적 또는 목표의 실질적 효과에 대하여 회의적인 시각을 갖는다는 것을 고려하면, 이들 회의론자에게 '큰 틀'을 제시하는 것이유용할 수 있다. 이는 교수의 역할은 '학생이 지식을 함양하고, 학생의 자기실현을 도와주며, 개인적 그리고 사회적 발전을 지향하며, 사회의 변화와 조직의 효과성을 추구해야 한다'라는 것이 될 수 있다.

When curricular modification is anticipated, renewal team members should recognize that many teachers in the trenches have difficulty understanding, let alone applying, grandiose concepts such as: “statements of intent”, “overall aims” and “general learning outcomes”. Many experts still favor development and application of clear, concise “objectives” which are acceptable to all or most of the teachers. Given that many teachers are skeptical about the benefits and practical utility of comprehensive goals and/or objectives, it may be useful to expose the skeptics to the overarching idea that teachers should strive to facilitate attainment of broad goals including: cultivation of the intellect; individual self actualization; personal and social improvement; social transformation and organizational effectiveness.

'성과바탕모델'을 맹목적으로 적용하는 것은 적절하지 못하며, '학교가 원하는 졸업생'의 모습을 달성하는데 어떤 어려움이 있는지를 드러내기 위해서는 교육과정 개선 초기에 격렬한 토론이 있어야 한다.

In a 1985 education publication describing overarching goals, the author suggested that health care teaching should aim to train potential health care workers to completely “undertake the responsibilities expected of them in professional practice” (Bandarananayake 1985). Thirty years later this aim still has merit. “Blind adoption” of an outcomes-based model may not be the optimal approach (Rees 2004). To obviate the problems inherent in what the school wants the graduates to look like, we recommend an early vigorous debate that leads to consensus on what the graduates should know and be able to do.

근거에 기반한 교육과정 개선을 추구하라

Tip 5 Promote an evidence-based approach to curricular renewal

기존의 교육 관련 문헌을 잘 연구하고 정리해야 한다. 그리고 어떤 교육과정 모델을 활용하든지 다음의 네 가지에 구성요소에 대한 토론이 반드시 필요하다.

Regeneration and enhancement of learning experiences must entail diligent ongoing review of the major health sciences education publications for articles on curriculum to determine what is most appropriate for individual contexts. The results of the review should then be summarized, written down and distributed to all program coordinators and teachers. Fixation on curriculum as a “planned educational experience that encompasses goals, teaching and learning methods and outcomes” may be difficult given the bewildering array of available models, each with detractors and proponents. Whichever curriculum model is favored, four structural elements should inform all curriculum renewal discussions: (1) curriculum content, (2) learning approaches, (3) assessment practices and (4) evaluation practices (Prideaux 2003).

여섯 가지 모델이 있다.

Among the curriculum models currently used in the health sciences, six dominate the education literature: 

(1) Subject or disciplined-based curricula (Grant 2006). 

(2) Problem-based curricula (Barrows 1996; Colliver 2000). 

(3) Experiential learning curricula (Kolb 1984). 

(4) Spiral curricula (Harden 1999). 

(5) Clinical presentation curricula (Mandin et al. 1995). 

(6) Outcomes based curricula (Rees 2004). 

교육과정의 큰 틀은 '배움'과 '학습자'가 되어야지 '가르침'과 '교수자'가 되어서는 안 된다.

The problem-based learning curriculum model (PBL) has gained a foothold in universities in many countries but there is surprisingly little concrete evidence demonstrating that the PBL approach is superior to others. The clinical presentation curriculum organizes clinical and basic science instruction around clinical problems. Whatever descriptive label is affixed to the curriculum, situated learning should be prominent (Mann 2011). Critical to the success of this approach is the development of a community of learners who work in the collective (Douglas & Brown 2011) while striving to become successful health care practitioners. Active student engagement in the community over time gradually leads to acquisition of knowledge, skills and values of a professional. The overriding theme is that a curriculum should be chosen for learning and learners, not for teaching and teachers. Good teachers can provide good explanations of information that help learners understand.

교육 방법, 교육 전략, 학습 자원을 철저히 분석하라

Tip 6 Critically review the teaching methods, educational strategies, and learning resources

강의는 점차 감소하고 있으며 다음의 것들이 늘어나고 있다.

Lectures play an important role in most schools because they are an effective, efficient way of transmitting information and integrating concepts (Matheson 2008). However they are quantitatively on a decline at many schools because they are prone to produce passive student involvement in the learning process, especially if the lecturers lack charisma and speaking expertise. In many schools, lectures have been replaced by small group-based learning activities, problem-solving workshops and self-directed learning. Case discussion groups, role-plays and group-dependent or independent learning projects have also gained favor because they feature active learner engagement and are conducive to the development of communities of learners. Other worthwhile learning initiatives driven by adult learning principles include those that provide opportunities to pursue special individualized student interests. Examples of these are: travel to different learning sites, community hospitals or clinics for the disadvantaged, and service in underprivileged communities or countries. The use of information technology and computer-based learning is the norm in most progressive medical schools as are on-line chat groups and learning modules to package content. The rapidity of technological change requires a lively ongoing overview and renewal of how the technology is being used. The renewal team members must regularly monitor the use of technology to assure that it is appropriately exploited while guarding against excessive reliance on expensive gimmicks.

중요한 분야 중 하나가 시뮬레이션 기술이다.

Last, but certainly not least, is the rapidly advancing field of simulation technology. Simulation is especially applicable to the teaching of procedural medicine, patient interviewing and professional behaviors (Scalese et al. 2008). These technologies have opened up vast new learning opportunities for students at all levels and should be carefully considered.

교육에 '만병통치약'은 없겠지만 환자중심의, 자기주도적, 학습공동체, 시뮬레이션 접근법 등을 잘 짜여진 강의로 보충하는 것이 한 의과대학의 '교육 메뉴판'의 전면에 등장해야 한다.

Although there is no “one size fits all” approach to teaching and learning, patient-based self-directed approaches, communities of learners, and simulation approaches, all sprinkled with some well-delivered lectures, should be prominently displayed on the “education menus” of a medical school.

타당한 교육이론으로부터 교육과정의 변화와 교육법과 학습법의 개편이 이루어지게끔 하라

Tip 7 Ensure that sound educational theories inform the transformation and renewal of teaching and learning methods

학습의 네 가지 측면이 중요하다. 첫 번째는 "개념의 의미를 이해하는 것이 학습에 크게 영향을 미친다"는 것이며, 두 번째는 "학습의 맥락"이 지식 습득에 중요하다는 것이다. 세 번째는 "processing specificity"로서 어떤 것을 어떻게 배웠느냐가 나중에 배운 것을 인출하는데 영향을 준다는 것이다.네 번째는 "기억은 어떻게 암기하느냐"에 영향을 받는다는 것이다.

Fortunately, in the education literature, there is an impressive body of easily digestible research on curricular design and teaching methods (Bleakley 2009; Mann 2011; Prideaux 2003). Convincing research demonstrates that students should be exposed to an environment which encourages self-reflective learning and self-critical learning. Four aspects of learning influence students’ ability to acquire and store new knowledge in memory in a retrievable manner. 

• The first aspect relates to the fact that “understanding the meaning of a concept strongly influences learning”. 
• Second, “the context of the learning” is critical to knowledge acquisition. Learning in a clinical setting provides exposure to a powerful influence, sometimes called “the romance of medicine”. 
• Third, “processing specificity” is critical. This implies that the manner in which something is learned will influence one’s ability to retrieve what is learned. 
• Fourth, “memory is significantly influenced by practice of the task of remembering”. In other words, expertise is a function of time spent on learning (Ericsson 2004). Awareness of these principles can help to guide the renewal process.

과도하게 사실적 지식을 공급하는 것을 지양하고 임상 시나리오를 활용한 개념을 익히는 것을 강조해야 한다.

Curriculum renewal team members must assure that the applied curriculum aggressively eschews fact overload while emphasizing learning of concepts and dealing with clinical scenarios which embody the concepts to be learned. Thus, it behooves curriculum designers to construct student experiences which promote deep learning and critical thinking. For example, a module on high blood pressure might include the presence of a real patient or a “paper case” outlining a patient’s medical story. Subsequently, the instructor would encourage group discussion focusing on the meaning of the disorder and its consequences for cardiovascular disease. Constructivist theory posits that such a clinical example will facilitate students’ creation of their own understanding of the pathophysiology and prognosis of high blood pressure. Discussion of the problem in the students’ community of learners will further enhance understanding and remembering (Mann 2011; Vygotsky 1978). This constructivist theoretical framework may lead renewal committee members to embrace the clinical presentation model.

교사의 선택, 교수개발, 평가와 보상을 위한 준거를 개발하라

Tip 8 Develop criteria for teacher selection, development, evaluation and reward

연구를 잘 하는 사람이 잘 가르칠 것이라는 암묵적 가정이 있었다.

There is a dearth of research on teacher selection for health care instruction. In the recent past, teaching in the early years of medical school fell to highly trained research scientists who had been recruited to the medical school because of their scientific prowess. There was a tacit assumption that skilled researchers would be skilled teachers. In the clinical components of the curriculum, specialist physicians in teaching hospitals would do the teaching on the inpatient clinical services or in the ambulatory care clinics. It was uncommon to have significant involvement of generalist practitioners in teaching settings. Similar teacher selection models have been applied in other health care disciplines including nursing, occupational and physical therapy.

최근들어 교육자의 역할이 보다 주목받고 있다. 또한 더 이상 보상 없이 교육하라는 것은 받아들여지지 않는다.

Recently, the educator role is demanding more attention among health care workers recruited for patient care in offices, clinics and hospital wards (Weatherhall 2011). Increased recognition of the importance of the teaching role has been accompanied by renewed faculty enthusiasm for staff, or faculty, development (McLeod et al. 2011; Steinert et al. 2010) and in remuneration for teaching. Further beneficial developments have been observed in teacher recruiting practices. During the last 15 years there has been a significant increase in interest in education courses, faculty development workshops, and medical education masters programs. Established health care professionals who are interested in teaching and education research are enriching their professional lives by acquiring education expertise in courses and masters programs. Furthermore, for both generalist and specialist physicians, teaching without compensation is no longer acceptable. Stipends are often symbolic in nature, but recognition of teaching prowess and success is reward in itself. Formal teacher awards, many with monetary supplements, are routine in many European and North American medical schools and are gaining favor in other jurisdictions. Course directors responsible for teacher selection should cast the net widely and downplay teachers’ “need for developing a teaching dossier for promotion”. Teachers should be selected because they are passionate, committed communicators who understand students’ needs and who are ready to adopt proven advances in education practices and education technology. Past approaches have entailed flurries of attention to replenishing the educational mission of health professional schools immediately following introduction of a new curriculum. That approach is no longer valid. Dramatic changes in education theory and learning environments today require ongoing repetitive renewal targeting all health care teachers and administrators. Educators must assume the responsibility of encouraging and facilitating renewal and university leaders should recognize and reward teachers at all levels.

적절한 내용과 학습경험의 순서를 구성하라

Tip 9 Assure appropriate content and sequencing of learning experiences

내용의 선정은 사회적 질병부담에 따라야 한다. 광범위한 델파이 조사를 할 수 있다. 상당한 저항에 부딪칠 것이다. 그러나 지난 한세기동안 학생에게 지워진 부담은 "carcinoma of the curriculum"이라는 말이 붙을 정도이다.

Selection of content to be emphasized must be informed by the known societal burden of disease. One appealing process for the selection of relevant curriculum content begins with development of a large inclusive and superfluous list of clinical problems known to exist in the population. The problem list can then be subjected to a broad-based Delphi process using participants representing both health care providers and recipients of health care (McLeod et al. 2004). The rapidly expanding knowledge base to which learners will be exposed requires that the curriculum focus on a limited number of important representative problems (D’Eon & Crawford 2005). Success in developing a manageable content list will require vigorous resistance to pressure from “experts” and “specialists” who frequently overestimate the importance of learning details of disease in their field. For over a century, information overload has burdened students in the caring professions. The tendency of curriculum designers to comprehensively cover the field has been derisively labeled “carcinoma of the curriculum” (Abrahamson 1978).

이상적인 학습은 교육과정 개편을 맡은 팀이 제한된 숫자의 중요한, 대표적 임상문제들을 선정하고, 기초의학을 전략적으로 조직해넣었을 때 이뤄질 수 있다. 또한 개편 팀은 정기적으로 교육 현장을 방문해볼 필요가 있다.

Appropriate sequencing of curriculum content is best served by a strategically planned mixture of basic science and clinical problems. One effective sequencing model sees representative clinical problems introduced early with applicable physiology and basics woven into the discussion of the clinical problems (Mandin et al. 1995). The other end of the curriculum spectrum entails early introduction to basic sciences followed by clinical problems (Davis & Hardin 2003). In the latter model, it is recognized that regular “hints of relevance to clinical practice” can foster enthusiasm for learning the basic concepts. In the opinion of many, optimal learning will be assured if renewal team members select a limited number of important, representative clinical problems and strategically interweave basic sciences into the description of those problems. In addition, members of the renewal team should, at regular intervals, conduct mandated on-site visits to the lecture theatres, small group teaching facilities and clinical teaching sites to monitor the content, sequencing, and integration of basic and clinical elements. Monitoring experiences and provision of constructive feedback will help instructors and learners alike.

다면적 학습평가 프로토콜을 계획하라

Tip 10 Plan a multidimensional learner assessment protocol

학습자가 문제해결을 얼마나 잘 할 수 있느냐는 그 문제와 관련하여 활용할 수 있는 지식이 얼마나 풍부한가에 달렸다.

Once the curriculum developers have produced a model of “the product” or type of student desired, it should be relatively straight forward to determine the success of the renewed curriculum. Fortunately it is known that a learner’s ability to solve problems is highly dependent on the availability of the knowledge relevant to that problem (Prideaux 2007). Students should be able to show what they know and can do, and formal assessment approaches are available for the tasks.

한 가지 평가방법으로는 충분하지 않다.

The recognition that “assessment drives learning” should inform the use of assessment vehicles in the health care setting. Students will take seriously the material on which they will be tested, so the assessment process should be informed by, and reflect the curriculum (Weatherhall 2011). There is a general consensus that no one assessment vehicle is totally adequate; thus a multidimensional process is required. The outcome of quality assessment research is that many schools, drawing from the business world, are moving to “360 degree” assessment of learners. (Allerup et al. 2007; Rees & Shepherd 2005).

총괄평가에 앞서서 형성평가가 반드시 필요하다. 지속적으로 평가방식을 감독할 필요가 있다. 

Summative judgment of the learners’ knowledge, skills and behaviors acquired as a consequence of the formal curriculum should be preceded by ongoing formative assessment. Many well-developed assessment processes are available to test learners’ knowledge, skills and behaviors. These processes should be part of both the summative and the formative student assessments (Cook 2010). Experience reminds us that the tenor and breadth of assessment practices need constant monitoring. In some disciplines, multiple choice questions dominate; in others, broadly-based judgments of knowledge skills and behaviors are used. Members of the curriculum renewal team are ideally placed to act as beneficent insurgents by regularly visiting the various teaching units to observe, comment on and renew the assessment processes.

교육과정 개편을 평가하라

Tip 11 Evaluate the curriculum renewal

교사나 학생을 위한 것일 뿐만 아니라 사회적 책무이기도 하다. 시작시점부터 평가가 고려되어야 한다.

Given the dramatic explosion of information sciences and technological advances, ongoing curriculum renewal should be a prominent activity in the health sciences. Health sciences educators frequently introduce new educational approaches and technology but they may fail to heed the “stop, look and listen” aphorism. Ongoing curriculum renewal should mandate that we evaluate the impact of any change we introduce. Not only is such an approach beneficial to learners and educators, it is also an important part of our mandate to be socially accountable for the changes made and for monitoring the outcomes of the changes. Although there are several useful frameworks to guide the evaluation process we are comfortable with Kirkpatrick’s curriculum evaluation framework (Kirkpatrick 1994) but others may suffice. This framework has four levels of evaluation, each with different emphases. Level 1, reaction, focuses on participants’ reactions to the overall experience. Level 2 involves learning. The learning can focus on knowledge, skills, behaviors and/or attitudes. Ideally “pre-post” judgments should be used at this level. Level 3 involves evaluation of use and application of what the participants have learned in the curriculum. At this level, educators must make difficult decisions with respect to when, what and how to evaluate. Level 4 is probably the most important but, it is also the most difficult outcome (or set of outcomes) to measure. Ideally, educators should decide what constitutes optimal results and what constitutes a positive impact. Experience reveals that educators rarely evaluate at this level. Wise health care educators incorporate a plan for intermittent curriculum evaluation from the outset of the project. Content renewal should not be an add-on or an afterthought.

Tip 12 학습환경이 학습에 크게 영향을 준다는 사실을 기억하라

Remember that the learning climate significantly influences learning

"교육과정이란 학교 안의 가치와 아이디어, 사람과 자원 간의 상호작용이다"라고 했다.

Medical school curricula usually devote considerable space to outlining the structure, content and expectations of the learners but rarely address the nature of the learning environment or the “climate” in which the curriculum will operate (Genn 2001). It has been suggested that the curriculum should be thought of as “an interactive process involving values and ideas, people and material resources in the school” (Johnston 1992). Those critical elements of medical school learning environments and the unique impact they have on student achievement may be regarded as the soul and spirit of the school. The climate involves not only the learners, but also the teaching and auxiliary staff whose well-being may have a significant impact on the lives and learning of students. For example, if teaching by intimidation and bullying are permitted, student morale and enthusiasm for learning will be suppressed. A climate dominated by overwhelming fact overload can seriously impair student morale and derail enthusiasm for learning.

다음과 같은 문화를 갖추어야 한다.

A renewed curriculum requires solid cultural influences as cornerstones for the development of a successful climate. The support stones include: 

• student-centered teaching; 
• teacher openness to criticism; and 
• a dedication to helping students develop independent self-directed learning behaviors. 

Student achievement is inextricably linked to students’ personal wellness and satisfaction. The school administrators, represented by the curriculum renewal team, must evaluate and monitor the learning climate and assure that negative influences are rooted out before they take hold and poison the environment. In this role, the renewal team members should have a high profile in the ongoing workings of the school and they should have the support of the Dean of Health Sciences.

Med Teach. 2015 Mar;37(3):232-8. doi: 10.3109/0142159X.2014.932898. Epub 2014 Jul 10.

Twelve tips for curriculum renewal.

보건의료인들 간 이메일 의사소통: 근거기반 가이드라인 (Academic Medicine, 2015)

Professional E-mail Communication Among Health Care Providers: Proposing Evidence-Based Guidelines

S. Terez Malka, MD, Chad S. Kessler, MD, MHPE, John Abraham, MD,

Thomas W. Emmet, MD, MLS, and Lee Wilbur, MD

이메일과 인터넷 활용이 광범위함. 젊은 층에서 더 많이 쓰고, 이들은 곧 의사가 될 것임.

As of January 2014, 87% of American adults use the Internet.1 A 2004 study of health care professionals found that 64% use e-mail to communicate with each other for work-related purposes.2 According to the Pew Research Internet Project, 18- to 29-year-olds are the most frequent e-mail users, so we can expect this figure to increase as more young physicians enter the workforce.1 Simultaneously, the Internet is becoming increasingly accessible with the rising prevalence of smartphones and portable handheld devices.3 As e-mail is now a frequent method of workplace correspondence, it is imperative for users to have an understanding of appropriate etiquette and proper professional e-mail use.

의사들은 다양한 용도로 이메일을 사용함

Physicians use e-mail for a multitude of purposes: 

• to obtain consults, both formal and “curbside”; 
• communicate with patients; 
• collaborate on scholarly projects; 
• perform administrative duties; and 
• conduct routine communication. 

이메일 기술은 원격진료 측면에서도 다양한 영역으로 확대되고 있음. 용이하면서도 적절한 이메일 활용이 중요함.

E-mail technology also expands the scope of telemedicine, allowing for remote consultation, radiographic assessment, and patient care. Facile e-mail use and appropriate e-mail communication skills are vital to this growing field.

사례

The Case for Evidence-Based E-mail Guidelines for Physicians

부적절한 이메일 사례

Imagine receiving an e-mail from a colleague. There is no subject line. It is written in all capital letters, uses abbreviations or slang terminology and poor grammar, employs questionable humor, and even includes sensitive patient details. Its origin is from a personal e-mail account, not a professionally affiliated account, and its signature line is more suitable for informal communications than workplace correspondence (see Box 1 for an example). Although this is an extreme example, such an e-mail demonstrates several features that may be deemed unprofessional, and even illegal, while highlighting the challenge of using e-mail in a professional setting.

의사와 환자 사이에서 중요한 이슈가 된다. 적절한 라뽀를 유지하고, 환자 자료를 전송하면서 보안과 비밀을 유지하는 것이 중요하다. AMA와 AMIA는 이러한 어려움을 깨닫고 가이드라인을 제시했다.

These issues are of tremendous importance when e-mail is used for communication between physicians and patients. Maintaining a professional rapport through electronic correspondence and ensuring the security and confidentiality of transmitted patient data are paramount concerns. The American Medical Association (AMA) and American Medical Informatics Association (AMIA) recognized these challenges and have published consensus guidelines for physician-to-patient e-mail use.4,5

그러나 아직 근거기반가이드라인은 없다. 의사의 특성을 반영한 이메일 에티켓에 대한 것도 없다. 

Despite the attention paid to physician-to-patient e-mail use, there are no evidence-based guidelines addressing the use of e-mail among physicians.3 Many online sources and lay publications promote guidelines for professional e-mail etiquette; however, these guidelines may not address the unique needs of physicians. Serious legal and ethical issues may arise when e-mail is used between physicians that are not fully covered by simple etiquette. Our goal is to briefly summarize the literature relating to professional e-mail use between health care providers, to discuss the challenges of e-mail use within health care, and to offer our recommendations for professional e-mail use.

가이드라인 개발 (문헌 조사)

Developing the Guidelines: A Literature Search

To inform our development of professional e-mail guidelines, a comprehensive search of the literature was performed by one of us, an experienced medical librarian (T.W.E.), using the following databases: Ovid MEDLINE, PubMed (for non-MEDLINE records), Embase, the Cochrane Library, CINAHL, PsycINFO, Communication & Mass Media Complete, and Google Scholar. Searches were conducted between October 3 and 12, 2012, and all databases were searched from inception. We also reviewed bibliographies of relevant studies for additional references. Database-specific subject headings and keyword variants for each of the three main concepts—electronic mail, communication, and physicians—were identified and combined (detailed search strategy available upon request). We limited results to the English language, but no other limits were applied. Ultimately, 4,185 titles and abstracts were independently reviewed by two of us (T.M. and J.A.) for relevance. We identified 15 articles that directly discussed interprofessional physician e-mail use: 9 editorials or commentaries and 6 journal articles.

입원환자에 관한 이메일 활용의 장점과 단점

Benefits and Drawbacks of E-mail Use in an Inpatient Setting

장점: 신속한 회신, 사용의 편이성

We identified three studies that surveyed physicians and nurses using e-mail for communication in an inpatient setting. O’Connor and colleagues 6 performed a survey of providers within an intensive care unit communicating by e-mail over an encrypted cellular network via portable handheld device. Wu and colleagues 7 performed a mixed-method assessment of nursing and physician communication via handheld wireless device on an inpatient medicine service. A third survey-based study by Singarella and colleagues 8 assessed physician opinions on e-mail use and also analyzed the content of e-mails sent. Across all studies, survey respondents reported that response times were more rapid and e-mail was easier to use than alternative methods such as written correspondence, phone calls, and paging. Over 90% of those surveyed by O’Connor and colleagues 6 felt that patient care was improved as a result of e-mail use because of the efficiency of e-mail communication.

단점: 환자 상태가 복잡한 경우, 두 차례 이상 메일이 오가야 하는 경우, 시급성에 대한 인식 차이, 면대면 의사소통 저하

Several negative impacts of e-mail use were also identified by survey respondents. E-mail was deemed to be less efficient when subject matter was complex, requiring more than one initial e-mail and reply. There was also frequent discordance between senders’ and recipients’ perceptions of a message’s urgency. Singarella and colleagues 8 noted that e-mail users uniformly assumed a more casual tone and were more apt to make grammatical errors than those communicating by telephone or in writing. The most frequently cited negative impact in these studies was a reduction in face-to-face communication which potentially weakened interpersonal relations.6–8

단점 추가: 보안, 이메일 내용이 EMR에 포함되는가의 문제

A further concern is the security of transmitted patient information. Although these studies explored the use of e-mail on handheld devices provided by the hospital, none commented explicitly on how these devices were encrypted and secured. None of the studies specify if the messages sent on these devices assume a formal place within the patients’ charts or electronic medical records (EMRs). Discussion of a patient’s condition with a “casual tone” or in an e-mail fraught with grammar errors is concerning if these e-mails are considered a part of the EMR. If e-mails are not included within the patient record, care must be taken to document these communications when they lead to a change in management plan or affect the patient’s clinical care.

프로페셔널리즘 이슈가 있다. 이메일의 톤, 시급성에 따라 적절한 타이밍의 회신, 언어적 의사소통의 감소

The question of e-mail use in the inpatient setting highlights both positive and negative elements of workplace e-mail. In the studies we reviewed, users of e-mail for communication within inpatient teams felt that e-mail improved their efficiency and had a positive impact on speed and ease of communication. However, issues with professionalism were identified in all three studies. Primary areas of concern were the casual tone of e-mail use, the lack of timely response to e-mails perceived as urgent, and the resulting decrease in verbal communication. We believe that these studies emphasize the need to maintain professional formality in workplace e-mail communications, and we suggest that e-mail should be avoided when the issue is complex or time-sensitive. In addition, when e-mail is used specifically for the purpose of communicating secure patient data, specific hospital guidelines should be in place to ensure the confidentiality of these transmissions and to address the appropriate documentation of these transactions within the patient medical record.

이메일 구성이 평판에 어떤 영향을 줄 수 있는가?

How E-mail Composition May Affect Professional Reputation

가장 부정적으로 인식되는 이메일 특성(배경색 있는 것, 폰트, 제목없음, 인사없음). 이는 발신자에 대한 부정적 인식을 유발하며, 회신도 더 늦게 하게 만든다. 적절한 제목, 형식을 잘 갖춘 첫인사와 마무리, 적절한 문법과 스펠링을 갖춘 이메일일 경우 발신자에 대해서 더 professional하다는 인식과 즐거운 감정을 느끼고 긍정적으로 인식하게 된다.

Our literature review also pointed toward further consequences of unprofessional versus professional e-mail use. A study of surgery residents published in the Journal of Surgical Education sent 100 e-mail examples to physicians in training.9 The most negatively rated features were the presence of a colored background, atypical fonts, lack of a subject line, and lack of a formal salutation. E-mails containing these negatively perceived characteristics were “likely to result in a negative perception of the sender and delays in response time.” Conversely, respondents were more likely to perceive senders as professional and pleasant when they sent e-mails that employed positively perceived features such as a descriptive subject line, formal greeting and closing line, and proper grammar and spelling.

이메일 구성에 얼마나 무관심할 경우 직업적 명성에 치명적일 수 잇음을 보여준다. 워딩과 형식은 수신자가 발신자에 대해서 느끼는 감정 뿐 아니라, 적절한 타이밍에 답장을 해줄 것인가에도 영향을 준다. 따라서 우리는 의과대학생들도 이러한 이메일 작성과 부적절한 메일을 받았을 경우 적절한 피드백을 제공하는 방법에 대해서 교육받기를 권고한다.

This study demonstrated that attention to e-mail composition is critical to professional reputation. E-mail wording and formatting affected not only the receivers’ perception of the sender but also the likelihood of a timely response. Therefore, we recommend that medical students receive training in composing professional e-mails and timely feedback when unprofessional e-mail characteristics are identified. Further, physicians should be aware of poorly regarded e-mail features and strive to avoid them in work-related e-mails.

형식의 중요성 

A Call for Formality

기본적 에티켓, 프로페셔널리즘, 환자정보 보안, 법적 문제 등에 대한 이슈를 제기했다. 우리가 리뷰한 대부분의 기존 연구들은 전문가의 의견이나 사례 등을 바탕으로 하거나, 상식적인 수준에서의 권고를 하고 있었다. 이러한 가이드라인들은 비지니스 문헌에서 언급되는 전문직으로서 공통적으로 따라야 할 이메일 가이드라인을 보여주며, 직관적으로 타당하다. 구체적으로 저자들은 직무와 관련된 이메일을 작성할 때 형식에 관심을 더 가질 것을 강조하고 있으며(high degree of formality), 보통 이메일이 지나치게 캐주얼하다고 지적한다.

The studies summarized above confirm the importance of appropriate e-mail use and raise crucial issues of basic etiquette, professionalism, patient confidentiality, and legal concerns. The majority of scholarly articles that we reviewed discussing appropriate use of e-mail are opinion based or anecdotal and offer commonsense recommendations for professional e-mail use that address some of the above concerns. These guidelines mirror the common professional e-mail use guidelines suggested in business literature and make intuitive sense. In specific, authors recommend maintaining a high degree of formality when using e-mail for work-related correspondence and note that current e-mail use is predominantly casual.

일부 구체적인 조언에서는 배경색을 넣지 않는다거나, 일반적이지 않은 텍스트 패턴, 약어, 이모티콘 등의 사용을 하지 않을 것 등의 형식을 강조하고 있다. 이메일은 발신 전에 문법과 스펠링에 어긋나지 않도록 퇴고가 필요하다.

We believe that observing these common-sense etiquette guidelines and erring towards formality in work-related e-mail communications is best practice. Some specific suggestions relating to formality include avoiding background colors, unusual text patterns, abbreviations, and “emoticons.” E-mails should be proofread for proper grammar and spelling prior to sending.10–17 See Box 1 for a revised e-mail that incorporates these recommendations.

HIPAA, EMR, 환자정보 보호

HIPAA, EMRs, and Protected Patient Data

보호받아야 할 건강정보라고 정의된 것이 있지만 HIPAA에서는 어떤 방법을 써야하는지 구체적인 언급은 없다. 여기서 모호한 지점이 발생한다.

The U.S. Department of Health and Human Services defines protected health information as all “individually identifiable health information” that is stored or transmitted in any form, including electronic.18 Persons or institutions who fail to ensure the confidentiality of protected health information are subject to criminal penalty. However, the Health Information Portability and Accountability Act (HIPAA) makes no clear specifications as to which privacy features (such as encryption software or secured networks) are considered adequate. This leaves ambiguity in the use of e-mail for transmitting patient data. For example, you may open an appropriately encrypted e-mail within your personal e-mail and then save that message to your inbox or to your personal laptop, or you may inadvertently forward an e-mail containing protected patient data to an unintended recipient. Smartphones and other personal handheld devices present additional concerns—for example, if you bring your hospital handheld device home with you in the evening or check your work-related e-mail in a public location. The privacy of even securely encrypted data is of concern in an age of hacking, computer viruses, and piracy.

AMIA와 AMA는 가이드라인을 만들었다.

The AMIA and AMA attempted to address some of these inconsistencies by establishing guidelines for the use of e-mail in physician-to-patient communication.4,5 Although these guidelines were not specifically targeted to physician to physician communication, they are the only available evidence-based guidelines that discuss the protection of electronically transmitted patient data. More and more physicians are using e-mail to discuss patient care or to share clinical information; therefore, these guidelines are pertinent to interprofessional e-mail use as well as communication between physicians and patients.2

AMIA의 권고

AMIA recommends that printed guidelines should exist within each practice that clearly detail the security mechanisms in place. No correspondence containing protected patient data should occur outside of these established security mechanisms. AMIA further suggests that e-mail should never be left open on a workstation screen, that e-mails containing patient data may never be forwarded without written permission from the patient, and that all e-mails containing patient data be clearly listed as confidential in the subject or top of the e-mail message.

경험적 사례들: 이메일은 보호장치 없이 의사들 사이에서 돌아다니고 있다.

Anecdotal accounts suggest that e-mail containing protected patient data is routinely being exchanged between physicians via e-mail with none of the above safeguards in place. All of us, for instance, have personally received unsecured patient information via e-mail during the preparation of this article. This is clearly a tremendous professional liability as well as a medicolegal risk. We recommend that every institution provide clear guidelines for acceptable methods of transmitting secured patient data via e-mail, based on the AMIA/AMA guidelines, and that physicians take every effort possible to ensure the security of patient data when discussing patients via e-mail.

또 하나의 불확실한 영역은 e-mail correspondence에 대한 것이다. 이메일을 통해 논의된 환자 진료 관련 결정은 EMR의 사각지대이다. 

An additional area of uncertainty is the documentation of e-mail correspondence. E-mail is routinely used within the workplace for obtaining consults, sharing interesting cases, providing changeover, and discussing general patient care.2 In the studies of inpatient team e-mail use, the transmitted messages were not necessarily included in the EMR. These undocumented conversations become a concerning “blind spot” within the EMR or paper chart if patient care decisions result from those interactions. In the event that these messages are included verbatim in the EMR, care must be made to ensure that the content is professional and formal in nature, which was not the case in the studies that observed inpatient team e-mail use.6–8 Additional research is indicated to fully understand the legal and ethical risks of using e-mail for the transmission of patient data and to provide guidelines for appropriate documentation of e-mail correspondence within the patient medical record—electronic or print.

이메일의 잠재적 불명확성 회피

Avoiding the Potential Ambiguity of E-mail

면대면 소통 없이, 표정 없이, 목소리 변화 없이 이메일로만 이뤄지는 의사소통에는 오해의 여지가 있다. 

Another concern noted throughout the literature and also addressed in the AMIA/AMA guidelines is the increased potential for misunderstanding when communication occurs primarily over e-mail. With no facial expression, vocal inflection, or opportunity for real-time clarification, elements such as humor may easily be misinterpreted, and an angry or firm e-mail may come across more harshly than intended.

이메일은 사용하기 쉬운만큼 감정섞인 이메일을 보내기가 쉬워진다.

A benefit of e-mail is that it is fast and convenient. The downside to this accessibility is that it is very easy to compose an emotional e-mail in the heat of the moment or a thoughtless e-mail carelessly that may not convey your intended message. The AMIA task force cautions that “irony, sarcasm, and harsh criticism should not be attempted in e-mail messages” because “the impersonal nature and ambiguity of e-mail often results in real or imagined exaggeration of animosity toward the recipient.”5

퉁명스럽거나 말만 번지르르한 이메일을 보내기가 쉬워진다. "다른사람을 얕보는, 경솔한, 차별하는 말이 담긴 이메일은 지웠다고 하더라도 다시 되돌아오게 되어있다." 이메일은 중앙병원의 저장장치에 저장되고, 감시당하고, 통신사에 의해서 회수될 수 있음을 인지해야 한다.

The casual nature of e-mail may also predispose clinicians to send offhand or glib e-mails. While unprofessional remarks are never appropriate in a work setting, there is an added danger to expressing these sentiments by e-mail. Comments that once would have been a casual aside now may be stored on a hard drive forever, or within a patient’s EMR, or forwarded in error to an unintended recipient. Kane and Sands 5 note that “‘deleted’ messages containing disparaging, flippant, or incriminating remarks have come back to haunt physicians.” It is important for clinicians to be aware that even personal e-mails may be stored on a central hospital hard drive, monitored, or available for retrieval by cellular service providers.

요약 

Summary of Recommendations

On the basis of analysis of the available literature as well as our personal observations, we suggest a formal set of evidence-based guidelines for the use of e-mail in a professional setting:

• 1. Proofread each e-mail for proper spelling, grammar, and punctuation.11,13,15–17
• 2. Use a meaningful subject line that is descriptive of e-mail content.10,12,13,15–17
• 3. Avoid background colors, patterns, all capitals, and unusual fonts.9–12,14,16,17
• 4. Avoid humor that may be misinterpreted.11–13,15
• 5. Don’t send an e-mail to the wrong person; be especially careful with reply all and mass forwarding.11,12,15–17
• 6. Don’t send emotionally charged e-mails; consider a direct conversation for complex or sensitive topics.11–13,17,18
• 7. Transmit protected patient data cautiously using a private or secured computer or handheld device via an encrypted, secured network. Avoid sending such data to or from a public e-mail service such as Gmail, Yahoo, or Hotmail.4,5

Concluding Remarks

The topic of e-mail communication between health care providers has been broadly discussed but, to our knowledge, is underresearched. Our review of the literature did not reveal any formal guidelines or curricula for e-mail use among physicians. Although e-mail is fast and convenient, this accessibility has led to a decrease in formality and increase in errors and unprofessional behavior. E-mail recipients form perceptions of e-mail senders based on the format, content, and tone of their e-mails. An e-mail that is perceived as unprofessional may be less likely to receive a response or may receive a different response than one that follows etiquette guidelines. In addition, serious medicolegal and ethical concerns arise when e-mails contain patient data or unprofessional remarks. The appropriate use of e-mail has the potential to affect one’s professional reputation and to influence clinical, and potentially legal, outcomes. The recommendations we make for interprofessional e-mail use are based on the literature review and analysis above.

Acad Med. 2015 Jan;90(1):25-9. doi: 10.1097/ACM.0000000000000465.

Professional e-mail communication among health care providers: proposing evidence-based guidelines.

교수개발프로그램의 전이(transfer)를 높이기 위한 12가지 팁(Medical Teacher, 2014)

Twelve tips for increasing transfer of training from faculty development programs

STEPHEN L. YELON, J. KEVIN FORD & WILLIAM A. ANDERSON Michigan State University, USA

Introduction

임상적으로 교수가 될 준비가 잘 되었다고 해도, 교수로서 맡아야 할 새로운 학문적 역할에 모두 준비된 것은 아니다. 교수개발의 정의는 아래와 같다. 교수개발의 수단으로는 멘토링, 자기주도학습, 웹기반학습, 워크숍, 펠로우십 프로그램 등등이 있다.

Physicians who choose a career in academic medicine are generally well prepared clinically to serve as faculty; however, they may not be as well prepared to be successful in their new academic roles (Steinert et al. 2006). Medical schools have provided training in essential academic skills through faculty development programs. Steinert et al. (2006) define the term faculty development as the “broad range of activities that institutions use to renew or assist faculty in their academic roles”. Medical schools have largely focused faculty development efforts on new faculty, addressing the knowledge, skills and attitudes necessary for instruction, scholarship, and administration (Bland et al. 1990). Faculty developers have taught these skills via mentoring, self-directed learning initiatives, web-based learning, workshops, and full or part-time fellowship programs (Steinert 2000).

미시간주립대학의 교수개발 프로그램 개요(1)

The Michigan State University’s (MSU) Primary Care Faculty Development Program is one example of a successful national on-campus/at-home, part time faculty development fellowship program that ran from 1978 to 2012. The overall goal of the program was to prepare new primary care physicians for full-time careers in academic medicine. The fellowship curriculum addressed the basic roles of instruction, research, and leadership and also included other domains such as the use of computer technology, and adjustment to working in the academic community. For each role, there were objectives for all participants to attain. For example, for instruction, fellows, as the program’s participants were known, were to plan and present a lesson, teach a psychomotor skill, give feedback, and carry out selected clinical teaching approaches.

미시간주립대학의 교수개발 프로그램 개요(2)

Six hundred and one academic physicians from medical schools and primary care residencies completed this fellowship. Fellows spent a total of four weeks at MSU learning and practicing teaching, research or leadership skills in workshops, seminars and small group activities. At their home institutions, fellows completed assignments and projects to practice new skills or to demonstrate their attainment of program objectives.

미시간주립대학의 교수개발 프로그램 개요(3)

The MSU faculty development fellowship program used both formative and summative evaluation strategies. Fellows consistently expressed high satisfaction with their learning. Generally, fellows stated they were able to master most, if not all, of the program learning objectives, and, subsequently, supervisors noted positive changes in fellows’ faculty behavior.

교육의 현장전이에 대한 효과를 평가하는 것이 어렵다. 현장전이(Transfer)의 정의와 영향을 미치는 세 가지 주요 요소(학습자 특성, 훈련설계방식, 근무환경)

One of the most challenging outcome evaluation questions for the MSU Faculty Development Fellowship Program was a question of transfer of training, that is, the degree to which fellowship program graduates used at their home institutions the methods of instruction they acquired. Transfer is defined as the extent to which trained knowledge and skills are applied to the work context. Systematic reviews of training transfer research have identified three major factors affecting the extent of transfer to the job: 

• trainee characteristics (e.g. trainee motivation to learn), 
• training design features (e.g. incorporation of learning principles), and 
• work environmental factors (e.g. supervisory support for training) (Baldwin & Ford 1988; Blume et al. 2010; Grossman & Salas 2011).

MSU 교수개발 프로그램의 평가결과 요약. 신뢰할 수 있고, 실용적이고, 필요한 것을 적용한다. 조직이 필요로 하고, 배운 내용을 적용하는 것에 대한 보상이 있고 그것을 지원받을 수 있기에 적용한다.

MSU fellowship program faculty systematically investigated the transfer of instructional skills from the MSU program over the past 15 years (Yelon et al. 1997 2004, 2013; Sleight & Reznich 2006). In brief, the researchers uncovered a continuous, dynamic, transfer process whereby individual fellows perceived the utility of a method learned in the program, became ready to use it, applied it, and learned from its application. Fellows decided to use a method when they perceived its credibility, practicality and need. Immediately, and over years, in different ways and in varied contexts, fellows applied behavioral skills such as presenting a lecture, teaching a psychomotor skill, and giving feedback, as well as intellectual skills, such as planning, analyzing, and evaluating instruction. Fellows reported continuing to use what they learned because their well-designed plans were effective in addressing recognized institutional needs, and because they were supported and rewarded for application.

'전이'란 복잡하고 점진적으로 나타나는 단계로서, 훈련 이전, 훈련 중, 훈련 후에 걸쳐서 나타난다.

Researchers reviewing transfer, including the articles about MSU fellows, proposed that transfer is a set of complex, gradually emerging processes taking place before, during, and after training (Ford & Weissbein 1997; Yelon & Ford 1999). Accordingly, faculty developers must strive to incorporate strategies and principles that are effective in facilitating transfer prior to, during, and following a training experience.

본 article의 목적

(...)

MSU 펠로우십을 기반으로 진행된 연구, 기존 논문, 이론적 논문, 33년간의 운영경험에 기반한 팁임을 밝힌다.

The tips are based on the research conducted by MSU fellowship faculty about the ways medical fellows transferred instructional skills they learned to their work, reviews of empirical literature on transfer (Baldwin & Ford 1988; Baldwin et al. 2009; Grossman & Salas 2011), theoretical articles on transfer (Broad & Newstrom 1992; Yelon & Ford 1999; Yelon & Sheppard 1999; Broad 2005; Ford et al. 2011) and knowledge we gained from thirty three years of experience in directing, teaching and evaluating the MSU Primary Care Faculty Development Program.

무엇을 필요로 하는지 분석하라

Tip 1 Assess development needs

프로그램의 내용이 참가자들에게 관련된, 효용성있는 것일 때 현장전이가 촉진된다.

Researchers assert that participants’ perceived relevance and utility of an instructional program’s content has a strong influence in producing transfer (Grossman & Salas 2011). For example, MSU medical fellows said they transferred what they learned in the program because they believed they needed the skills taught to do their work well (Yelon et al. 2004, 2013). One fellow, who needed to improve his medical skills course reported that he knew immediately that he would apply one of the program’s instructional goals: able to systematically teach psychomotor skills. Another resolved to apply lesson-planning skills after practice when he realized he could do the same with his students.

프로그램 설계는 요구사정으로부터 시작한다.

Thus, begin designing a transfer-oriented faculty development program by conducting a needs assessment of faculty participants (Grant 2002). For example, program planners may investigate what potential participants do to meet clinical and academic expectations. Planners may search for the competencies that participants need to be successful in those academic roles, and, especially the competencies they lack. Fortunately, several researchers have provided, as a head start, lists of essential faculty competencies to consider (Bland et al. 1990; Harris et al. 2007; Srinivasan et al. 2011).

'요구되는 역량'이 프로그램의 목표가 된다.

Needed competencies become program goals: the statements that learners will use to judge personal relevance, and that program staff will interpret to design transfer-oriented, learning experiences. In addition, program planners can take into account the work conditions and constraints they find that are likely to affect participants’ attempts to transfer new skills after the program.

요구조사를 위해 활용할 수 있는 방법

Investigative strategies include direct observation, surveys, focus groups, and interviews. For example, one may ask participants on a survey to assess their current level of competence for each academic medical skill, as well as their interest and need to learn more. Supervisors may add their observations as well.

프로그램을 지원하면서 어떤 기대를 하고 왔는지 알아보기

Tip 2  Communicate the expectation of application

참가자들은 학습한 내용을 성공적으로 활용한 근거를 제시해야 한다.

To foster transfer, convey to participants that they are to apply at work what they learn from the program. Broadcast this message at every possible opportunity: improvement of academic medicine requires application of knowledge, skill and attitude; and application takes time, hard work, and a little help from friends. Specifically, state that session attendance or participation alone is not acceptable. Participants must provide evidence of the successful application of learning.

모든 세션을 시작할 때 실제 상황과 관련될 수 있는 학습목표를 언급하면서 시작하는 것이 좋다.

For instance, program staff may remind participants of the need to apply, by starting every session with specific objectives that are related to real world roles. They may communicate expectations of transfer in their exposition as: “Here’s something you can use when presenting at noon conferences as I did when…” Staff may continue by conducting simulation exercises of that performance. Further, to account for differences in participants’ duties and work environments, one may convey, by statement and example, permission to adapt methods to their specific circumstances.

적용시에 필요한 지원을 해줄 것을 약속하기

Tip 3 Secure support for application

Support의 의미

The term support, in the context of transfer, refers to the encouragement and the time and physical resources given to participants to motivate and enable them to apply what they learn. Researchers identify support as a crucial factor in promoting transfer, both in the short and the long run (Baldwin & Ford 1988; Broad & Newstrom 1992; Sleight & Reznich 2006; Grossman & Salas 2011; Yelon et al. 2004, 2013). Those participating in a faculty development program need the support of their colleagues and especially of their supervisors.

프로그램에 참가하는 사람들은 자신의 일상업무를 벗어나서 상당한 시간을 할애하게 되며, 동료로부터의 저항에 부딪칠 수도 있다. 따라서 프로그램을 시작하기 전에 참가자와 참가자의 상관, 프로그램 관리자로부터 공식적 동의(시간 허용, 새로운 접근법의 활용 등에 대한 동의)를 받아두는 것이 좋다.

Participants in lengthy programs must spend considerable time away from their everyday duties. Thus, participants may encounter resistance from colleagues, and they may become discouraged from spending time applying what they have learned. Therefore, before the onset of the program, gain formal agreement from supervisors and administrators for released time and for allowing attempts at perfecting new approaches, both during and after the program. Consider a phone or live conversation about needed support between a participant, a supervisor and a program staff member, followed by a signed document attesting to the aid promised. As to support from coworkers, teach participants how to gain and maintain help and encouragement from colleagues within and outside their own institutions. Monitor adherence to the agreement by inquiring and intervening if needed.

교수자가 현장전이에 대해서 가르칠 준비가 되어있어야 한다.

Tip 4 Prepare instructors to teach for transfer

전이에 영향을 주는 두 가지 주요한 요소를 고려해야 한다.

To prepare to teach for application, instructors in a faculty development program have to take into account two major factors influencing transfer, that is, participants’ personal characteristics and work environments (Baldwin & Ford 1988). Program instructors also must be able to design a program to create instruction that learners need, will be able to use and will accept – the third major factor influencing transfer.

청자가 무엇을 원하는지, 무슨 의무를 지고 있는지, 근무환경은 어떤지 알아야 한다.

First, instructors must know their audience’s needs, duties and work conditions. To gain that knowledge, instructors can study the results of their program’s needs assessment. To collect specific information relevant to their instruction, instructors may contribute questions to the assessment.

참가자가 그러한 조건에서 활용할 수 있는 방법이 무엇인지 알아야 한다.

Second, instructors must know methods that their participants could use, given their needs and work context. Hence, based on their research and experience, instructors have to know an assortment of practical, effective methods that can be broadly applied or specifically adapted.

참가자는 교수자가 신뢰할 만 하고, 그래서 교수자가 제시한 방법을 받아들일만 하다고 인식할 수 있어야 한다.

Third, participants have to perceive their program instructors as believable so as to accept the methods they propose. As a case in point, MSU fellows reported that they decided to apply ideas taught because they believed the program faculty when they said the methods would be effective (Yelon et al. 2004). To create believability, instructors could, for example, reveal their relevant professional experience and accomplishments when introducing themselves and using personal examples and cases. Further, to show their extensive knowledge of a topic, instructors could simplify what might otherwise be complex phenomena.

원칙과 방법을 모두 가르쳐야 한다.

Tip 5 Teach principles and methods

방법과 원칙의 차이

Methods are the ways of doing something, while principles explain how or why something works or happens. For example, at the start of a lesson, an instructor may use the motivational method of making a statement or presenting a case to show participants when they will use the lesson’s content and what they will accomplish. In contrast, the principle of meaningfulness explains why and how the motivational method works: to motivate students to learn, relate lesson content to the learners’ experiences, interests and aspirations (Yelon 1996).

원칙을 배움으로써, 방법이 작동할 것임을 확신할 수 있고, 방법의 절차를 익혀서 현장에서 적용할 준비를 할 수 있다.

Yelon et al. (1997) reported that before transfer, fellows in the MSU program engaged in two processes: (1) deciding to use a method and (2) preparing to use it. By learning principles of instruction, fellows were convinced that methods would work. By learning the methods’ steps, fellows were preparing to transfer.

원칙을 먼저 가르치고, 이를 바탕으로 논리적으로 방법을 선택할 수 있게 할 것을 권한다.

We recommend teaching principles first so participants will make logical choices of methods. 

• Accordingly, program planners select and define principles that explain how or why the methods to be taught should work. 
• Program instructors explain how using those principles will be of benefit: to justify a method, adapt the approach, trouble-shoot unsuccessful attempts, and create new techniques. Instructors illustrate each principle using cases set in familiar work contexts, and in each case, clearly accentuate the cause, relationship, and effect. After an explanation of each principle, instructors demonstrate an application and ask participants to explain how the principles were operating.
• Then, instructors teach methods that fulfill needed competencies, such as how to create a motivational attention-getter according to the meaningfulness principle. After justifying each method’s relevance and utility, instructors describe and demonstrate its steps, and provide practice and feedback.

적용에 대한 동기를 부여하기

Tip 6 Motivate to apply

실용적이고, 효과적이고, 필요성을 느껴야 한다.

Yelon et al. (2004) reported that fellows in the MSU program decided to apply what they learned about teaching when they perceived the content as practical, effective, and needed. Accordingly, learners will decide to use an approach when they believe: “I can employ this method”, “I believe this method will work” and “I must use this method to get what I need” (Yelon et al. 2004, 2013). For example, learners might decide immediately to use the One Minute Preceptor approach if they see it as easy to use, likely to work, and as likely to fulfill an important clinical teaching need.

가르칠 때 근거중심의, 충분한 이유가 있는 기법을 가르쳐야 하고, 적용 사례를 제시해야 한다.

To produce perceptions of credibility, practicality and need, program faculty must describe evidence-based and meaningful reasons for use of methods taught and must provide examples of application. Faculty must demonstrate methods fluently and clearly, present convincing research evidence of efficacy and efficiency, and provide realistic, challenging, active practice leading to success. For example, one technique to promote application is to show where, when, and how learners are likely to make use of the methods to be learned. Another is to show what the rewards will be for proper use and what the unfortunate consequences will be for non-use or improper use.

배운 내용을 상기시키고 적용할 때 필요한 도구를 제공하기

Tip 7 Provide tools to aid recall and application

실제로 적용할 때 까지는 오랜 시간이 걸릴 수도 있다.

Yelon et al. (1997, 2004, 2013) found that participants in the MSU program applied new ideas immediately, and weeks, months, and even years later. Fellows spoke of using instructional aids that converted to job aids, to provide reminders and offer specific guidance to apply. Fellows used program session notes, slides, checklists, decision aids, charts, lists of principles or skill steps, diagrams of processes, annotated models, and mnemonics.

For example, consider how the following combination of a novel name, a clear definition, a diagram, a poem, an outline of a plan, all on a card, served to help fellows remember and apply what they learned. In teaching goal-oriented instructional design, an instructor whispered that he was about to reveal the secret of instruction design. Projecting a definition and diagram, he gradually revealed each element of a consistent instructional system. He illustrated each part, demonstrated application, and then asked learners to apply it. He included the diagram and an outline of an instructional plan on a wallet-sized card. Later he added this poem:

• Real-world performance is the goal.
• The objective describes the test.
• If all the parts are consistent,
• your instruction will be the best.

참가자들이 직접 교육 설계를 하게될 때, 교육전략과 도구를 적절히 혼합해서 사용함으로써 원칙이 무엇인지 기억할 수 있고, 아이디어를 재구성할 수 있으며, 카드를 활용하여 계획을 짤 때 필요한 단서를 연결시킬 수 있다.

When fellows needed to design instruction, the combination of instructional strategies and transfer tools helped them remember the name of the principle, reconstruct the idea itself, associate certain cues that led them to the card, which they used to plan. For example, working at home, one fellow was planning to teach other physicians how to be successful on foreign medical missions. As she was thinking about what she wanted to teach, she remembered, “Real world performance is the goal!” Then she remembered the card, retrieved it from her home office, and applied the secret of instructional design to guide her creation of a coherent and effective session.

어떻게 적용할 수 있는가를 직접 보여주기

Tip 8  Demonstrate application

Yelon et al. (2004, 2013) found that MSU fellows made up their minds to apply methods when observing demonstrations and subsequently applied the methods that faculty showed them. Those results correspond to research showing that behavior modeling is a potent factor in producing transfer (Grossman & Salas 2011). To practice efficiently and to be able to apply at work, participants must see demonstrations of desired methods.

단순히 어떻게 하는지 보여주는 의미 뿐 아니라 다른 사람이 하는 것을 관찰함으로써 학습자들은 좀 더 집중할 수 있고, 다른 사람의 시연에서 가장 중요한 지점이 어디인지 집중할 수 있고, 행동을 따르기 위한 정신틀(mental template)를 형성할 수 있다.

However, an effective demonstration is more than merely showing participants how to perform. According to Bandura (1977), to learn and transfer through observing others, learners have to be motivated to pay attention, have to be focused on the most critical aspects of the behavior shown, and have to form a mental template to guide action. 

• "곧 활용하게 될 것입니다"라고 언급함으로써 동기부여 Thus, in demonstrations, motivate learners by stating that they will practice the procedures shortly, and will also use the procedures at work. 
• 보여주기 전에 어디에 집중해야 하는지 전체 단계의 절차를 알려줌 Next, focus attention on listed steps by telling learners what to look at and look for before the demonstration. 
• 시연하는 동안 참가자의 관심을 중요한 부분으로 집중시킴 Then, as demonstrating, direct viewers’ attention to the method’s important features. 
• 직접 해보도록 하기 전에 학생들로하여금 기억해야 할 단계를 수행해보도록 함 Finally, before providing practice, ask students to commit the steps to memory (Yelon & Maddocks 1987).

실습 기회 제공하기

Tip 9 Provide authentic practice

학습자들이 '이걸 사용할 수 있겠어' 라고 말할 수 있어야 한다. 

In studying the dynamics of transfer from a faculty development program, Yelon et al. (2004) reported that, to transfer, graduates need to be able to say, “I know I can use this notion”. The researchers noted, “Fellows gained their knowledge from workshop explanations and demonstrations, but were most affected by practice” (Yelon et al. 1997).

전체 시간 중 1/4~3/4 정도는 실제 연습에 할애해야 한다.

Thus, it is imperative to arrange time for practice opportunities. As a rule-of-thumb, practice should account for a quarter to as much as three quarters of instructional time, allotting more time as learning progresses.

한 가지 기본 원칙은 - 전이를 향상시키기 위해서는 - 실제 근무현장과 유사한 맥락에서 연습해봐야 한다는 것이다. 각 연습은 약간 어렵지면 궁극적으로 달성할 수 있을 정도의 난이도여야 한다.

One basic principle of transfer is – to promote transfer, learners should practice performance in contexts similar to those at work (Thorndike 1913; Grossman & Salas 2011). Thus, practice should simulate conditions at work. Each practice should be a bit more challenging, yet achievable. To prepare learners for success, precede practice by careful explanation, a complete demonstration and precise instructions and criteria.

프로젝트를 할 것을 요구하기

Tip 10 Require a project

프로젝트의 정의

Researchers reported that after the MSU program, fellows continued to develop their fellowship projects at work, such as developing a course, conducting research, or designing college policies (Yelon et al. 2004, 2013; Sleight & Reznich 2006). A project, in the context of a faculty development program, is an assignment requiring participants to apply skills learned to meet an important need at their home institution. The project is one instance of long term, development work expected of an academic physician. For example, in the MSU fellowship, participants are required to choose a significant curricular, research, or management problem at their home institution, to investigate and to design a solution, and then to implement the solution, evaluate it and report what they find.

교수개발 프로그램의 전이를 위한 완벽한 처방이다.

A project is the perfect prescription for transfer from a faculty development program. In fact, as they progress in developing their projects, participants are transferring skills learned in the program. In addition, a project incorporates several main factors affecting transfer: participant’s needs and interests, institutional needs, and a requirement that participants perform under real world conditions (Baldwin & Ford 1988). Specifically, participants choose projects of personal interest in collaboration with their home supervisor and the faculty development program staff. Together they design a feasible and useful project for their institution and perhaps the field at large.

발달과업(developmental work)는 프로그램 기간 내내, 그리고 그 이후에도 이어진다. 

The developmental work takes place over the length of the program, and it is designed to continue beyond. Just as it would in participants’ careers, the work leads to a substantial professional product. For instance, some fellows in the MSU program produced a course manual including a justification of the need for a course documented with a literature review, followed by instructional plans, materials, a report of a pilot test, and plans for revision and continued development. In addition, fellows wrote a paper in publishable form, created and presented a poster, and summarized their work at a conference attended by work supervisors.

그러나 단순히 프로젝트 수행을 요구하는 것 만으로는 부족하다. 참가자들은 자기평가, 지속적이고 광범위한 가이드와 피드백이 필요하다. 이를 위해서는 3~5명의  참가자를 프로젝트 그룹으로 묶거나, 한두명의 프로그램 스테프가 멘토 역할을 해주는 방법도 있다.

But, merely requiring a project does not insure successful accomplishment. To complete a project successfully, participants need frequent opportunities for self-assessment, and continuous, extensive guidance and feedback from others at every step of the project. That form of systematic supervision is made possible by project groups consisting of three to five participants and one or two program staff, who act as mentors. The purpose of project groups is to continuously promote, monitor, and support successful project development.

MSU의 멘토들은 자주, 그렇지만 타당한 데드라인을 정해준다. 

MSU mentors set frequent, but reasonable, deadlines for reviewing work. They encouraged participants to help each other succeed in transfer. Both mentors and group members provided feedback to improve, encouraged participants to advance, pointed out roadblocks to transfer and provided strategies to overcome the obstacles in work conditions. Participants provided support and pressure as they affirmed proper application and pinpointed errors. When the group formed a consensus about a participants’ work, it had a profound effect on that participant’s plans.

피드백 사이클을 만들라

Tip 11 Establish the feedback cycle

Yelon et al. (1997, 2013) described a continuous cycle of learning from transfer. Through practice and subsequent feedback in the MSU program, participants became willing and competent enough to try new methods at work. At home, they observed and analyzed the consequences of their attempts, were encouraged to continue to use successful methods, to improve unsatisfactory performance, and to try again.

이 사이클을 만들기 위해서는 참가자들에게 어떻게 유용하면서, 서로를 존중하면서, 솔직하면서, 긍정적 피드백을 줄 수 있을지를 가르쳐야 한다. 또한 어떻게 피드백을 찾고, 모아서, 받아들이고, 활용할 수 있는지도 가르쳐야 한다.

To begin to establish the continuous cycle of learning from transfer, teach participants how to provide useful, respectful, frank, and positive feedback to their own medical students and to each other. Then, for each practice in the program, ask learners to assess their own work and to check peers’ work before program staff makes their analyses. Next, teach how to seek, gather, accept and use feedback to improve themselves at home, as they will and must do the rest of their careers.

시작단계에서는 루틴한 그룹 피드백을 진행하는 것이 좋다. 그 다음으로 유사한 코멘트를 동료와 교수들로부터 얻도록 한다. 마지막으로 무엇을 들었고, 어떻게 할 것인지 보고서를 제출하게 하라.

• As a first step, establish a group feedback routine. Specifically, immediately after completing a task, ask the participant performers themselves to state what they did well and what they would change. 
• Next call for similar comments from peers and faculty. If performers become defensive, remind them to listen and attend to what they should continue to do and what they need to do differently to meet the criteria. 
• Finally, ask performers to report what they heard and what they would do the next practice round.

Subsequently, provide opportunities for participants to apply what they have learned about giving, listening to and accepting feedback. When practicing new skills or when presenting portions of their projects they need to listen to peers and staff in the program, colleagues and supervisors at home, and critics and experts outside their institution.

전이를 평가하라

Tip 12 Evaluate transfer

Over the years MSU fellows mentioned that they had used what they had learned. Faculty became curious about what, when, how, and why fellows had transferred what they learned. Consequently, MSU program faculty set out to evaluate transfer. Their purpose was to generate hypotheses about what had transferred and why, so as to improve the program and to inform colleagues about transfer. In the three studies mentioned (Yelon et al. 1997, 2004, 2013), Yelon et al. asked fellows to tell stories about their applications. From the stories faculty gained considerable food for thought about specific aspects of the program, and about transfer in general. One outcome was the tips you have been reading.

교수개발 프로그램을 개발하는 사람들이 전이를 촉진시키고 싶다면, 일부의 시간은 현장적용을 평가하기 위해 남겨두어야 한다. 온라인 설문이나, critical incident report를 제출하도록 할 수 있다. 

Thus, if faculty development program developers design a program oriented for transfer, they should consider allocating some time and effort to evaluate application. Evaluators can use qualitative research approaches calling for open-ended responses as recommended by Ford et al. (2011) to gain insight into the process of transfer. They can also use online questionnaires and surveys or critical incident reports using a social media site.

Conclusion

배운 내용을 적용하기 위해서는 (1)배운 내용을 수행할 능력이 있어야 하고 (2)시도해보고자 하는 의지가 있어야 하고 (3)자신의 시도가 지지받는다는 기분을 느낄 수 있어야 한다.

For faculty development participants to be likely to apply the methods they learn, they must: (1) be able to perform, (2) be willing to try and (3) feel supported (Mager & Pipe 1997). In our 12 tips, we account for each of these prerequisites to transfer. 

• First, to create the ability to transfer, participants learn needed competencies, as knowledgeable faculty guide them via relevant content, aids, demonstration, practices and feedback. 
• Second, to create willingness to apply, participants learn to adhere to a norm of applying the practical, needed approaches taught. As they achieve success and notice the effectiveness of their new skills, they want to try again. 
• Third, to produce support for application at work, participants learn to arrange for the physical resources, time, and opportunities to transfer. 

추가적으로 참가자들이 자신들의 새로운 시도의 효과에 대해서 건설적 피드백을 찾고, 모으고, 활용함으로서 스스로를 도울 수 있다면 더 안전함을 느낄 것이다. 또한 이를 위해서 이 12개의 팁은 하나의 시스템으로 활용되어야 한다.

Further, when participants realize that they can take care of themselves by seeking, gathering and using constructive feedback about the effects of their attempts, they feel secure. Because all three outcomes are necessary to influence transfer, the tips are to be used as a system. Clearly, designing a faculty development program to produce able, willing and secure medical educators is a demanding task, but well worth the effort.

Twelve tips for increasing transfer of training from faculty development programs.

PBL과 TBL 중 반드시 선택을 해야 할까? 둘의 장점을 결합해보자! (Medical Teacher, 2015)

Should we choose between problem-based learning and team-based learning? No, combine the best of both worlds!

DIANA DOLMANS1, LARRY MICHAELSEN2, JEROEN VAN MERRIE¨ NBOER1 & CEES VAN DER VLEUTEN1

1School of Health Professions Education (SHE), Maastricht University, The Netherlands, 2Department of Management,

University of Central Missouri, Warrensburg, USA

Introduction

변화는 사회와 보건의료의 항구한 특성이기에, 끊임없는 교육과정의 개선은 흔한 의과대학의 풍경이다. 20세기 경영학 교육에서 처음 기원한 TBL은 지난 10년간 보건의료 교육 영역으로 들어와 빠르게 퍼지기 시작했다.

Change being a stable characteristic of society and health care, continuing curriculum renewal is a familiar process in medical schools. In fact, medical schools all over the world regularly transform or modify their curricula to offer students optimal preparation for their work in the changing world of the health professions. Team-based learning (TBL) is presented as an attractive instructional approach especially for the acquisition of teamwork skills. Originating in business education in the 20th century, in the past decade TBL has made its entrance in health sciences education where its use is spreading rapidly Haidet et al. (2012).

1960년대에 시작된 PBL은 전 세계의 의과대학에서 도입한 혁신적 교육법이었다. 교수중심 교육을 학생중심 교육으로 바꾸게 되면서 교육과정 개혁의 핵심 돌파구가 되었다. PBL은 이미 많은 의과대학에서 활용되고 있으며 TBL은 교수법으로서 입지를 구축하고 있다.

Developed in medical education in the late 1960s, problem-based learning (PBL) was heralded as an innovative educational approach and adopted by medical schools all over the world. As a student-centered as opposed to a teacher-centered approach, it was a major breakthrough in curriculum reform (Frenk et al. 2010), and many medical schools embraced it as an alternative to the then dominant discipline-based, teacher-centered approach, characterized by content that was less relevant to practice, a strong emphasis on one directional transmission of knowledge and not enough emphasis on clinical reasoning and problem solving. PBL is used in many schools today, and TBL is strongly advocated as an instructional approach that is eminently suitable to prepare students for effective collaboration (Frenk et al. 2010).

PBL과 TBL의 공존은 몇 가지 질문을 던진다. 

The co-existence of PBL and TBL in medical education raises several questions: 

• What exactly are the differences and similarities? Is TBL the new breakthrough in education? 
• What can we learn from these two instructional approaches? 
• And, even more importantly: how do both approaches fit with current instructional design principles? Are there benefits to be gained from combining these approaches’ unique strengths? 

PBL과 TBL의 많은 변형이 있기 때문에, 최고의 교육적 효과를 내기 위하여 두 가지를 합하는 것이 가능할 것인가를 점검하기에 앞서서 각각의 특징을 알아보고자 한다.

Since there are many variants of PBL and PBL is understood and practiced differently in many institutions (Taylor & Miflin 2008), it seems logical to first describe our main characteristics before moving on to an examination of the possibilities for combining elements of PBL and TBL to maximize their joint educational powers (Table 1).

PBL의 주요 특징 

The main characteristics of PBL

주요 특징은 다음과 같다. 

PBL is a student-centered approach in which problems are the stimulus for learning. It is characterized by: 

• (1) learning through problems, 
• (2) small group sessions, 
• (3) group learning facilitated by a teacher, and 
• (4) learning through self-study (Barrows & Tamblyn 1980; Barrows 1996; Hmelo-Silver 2004). 

모든 학생이 직무와 관련된 동일한 문제를 놓고 토론함. 한 명의 교사가 촉진자 역할을 함. 문제를 논의하는 동안 더 공부할 이슈를 만들어냄. 개별학슴동안 모든 학생이 동일한 주제로 공부함. 개별학습 다음에 다시 모여서 개별학습에서 공부한 내용을 논의하고 서로 합함. 학생들은 무작위로 배정되고 6~10주동안 같은 조로 유지됨. 주당 2회, 2시간 정도 만남. 6~10주가 지나면 그룹은 다시 구성됨. 강의는 제한됨. 강의는 개별 학습 이후나 최종 토론 이후에 배치됨.

The problems are professionally relevant and discussed in small groups in which all students work simultaneously on the same problem. One teacher facilitates each group. During the initial discussion of the problem in the group, students generate issues that need further self-study. All group members study the same set of learning issues during individual self-study. After self-study the group reconvenes to discuss the findings from self-study and synthesize what they have learnt. Students are randomly assigned to these groups (six to ten students per group) and usually stay together for a period of six to ten weeks during which they meet twice a week during two-hour sessions. After six to ten weeks the groups are reformed. A limited number of lectures are scheduled. These lectures are scheduled after individual self-study and after the final discussion in the group.

사전에 논의된 문제를 토론하면서, 기존에 가진 지식을 활성화시키고, 자신의 지식과 문제 사이의 간극을 확인하고, 이 간극에 대한 자율학습을 한다. 자율학습과 최종 토론을 하는 동안 새로운 내용에 노출된다. 최종토론을 하면서 학생들은 다른 구성원이 공부해 온 내용을 접하면서 불확실했거나 자신이 아느느 것과 반대되는 내용을 확인한다. 교사는 촉진자의 역할을 하면서 핵심적 질문을 통해 토론을 장려한다. 타당한 추론을 하고, 인지적 불일치를 건설적으로 토론하고, 새로운 지식을 자신의 언어로 표현하고, 새로운 지식을 적용하고, 질문을 통해 문제를 해결하는 통찰을 얻는다.

By discussing a pre-set problem students activate their prior knowledge, identify gaps in their knowledge vis-à-vis the problem and use these gaps to generate issues for self-study. During self-study students are exposed to new content and during the final discussion after self-study. During the final discussion, students listen and explain to other group members what they have learnt, discuss issues that were unclear or contradictory and apply their knowledge to the pre-set problem. The teacher plays an important role as facilitator by encouraging students to ask critical questions, engage in sound reasoning, constructively discuss cognitive disagreements, explain new knowledge in their own words and apply new knowledge and insights to solve the problem in question.

TBL의 주요 특징

The main characteristics of TBL

TBL은 학습자 중심, 교수 지도의 교수법으로서, 모든 교실의 학생들이 5~7명의 소그룹으로 나누어 현실적 문제를 해결하게 된다. 한 명의 교사가 모든 팀의 촉진자 역할을 하며(20개 이상의 팀일 때도 있음), 그룹 세션이 있기 전에 학생들은 의무적으로 읽기과제를 하고 MCQ를 통해서 자신의 지식을 점검한다. 

TBL is a learner-centered, teacher-directed instructional approach for entire classes of students who are divided into small teams of between five and seven students to solve authentic problems (Michaelsen et al. 2008; Parmelee & Michaelsen 2010; Parmelee et al. 2012). One teacher facilitates various small teams, 20 or even more. Before group sessions students have to complete a mandatory reading assignment and their individual knowledge of the materials is assessed in a multiple-choice test. 

팀 단위에서 학생들은 개개인의 정답을 다른 팀원들의 답과 비교하면서 즉각적 피드백을 받는다. 이후 교사는 학생들이 어려워한 시험에서 나온 개념을 설명해준다. 학생들은 같은 팀 내에서 직무과 관련된, 보다 복잡하고 어려운 문제를 논의하며 동시에 자신들의 토론 결과를 다른 팀에 밝히게 된다. 교사는 전체토의 시간에 서로 다른 팀들이 서로 다른 답을 가지고 토론하는 과정에서 촉진다 역할을 한다. TBL에서 팀 내의 토론은 자율이나 팀 간 토론은 교사가 촉진자 역할을 한다. 팀 내에서 학생들은 서로 피드백을 주고받는다. TBL의 주요 특징은 다음과 같다.

In the teams, students discuss their individual answers to the exact same test items with each other to reach team consensus and receive immediate feedback on the team answers. Thereafter the teacher clarifies the concepts related to the test questions that students struggled with. Students then work in the same teams to resolve professionally relevant, challenging and complex problems and simultaneously reveal the solution or decision they have reached to the other teams in the class. The teacher facilitates the discussion in which the different teams challenge the decisions of other teams and defend their own decision in a plenary session. In TBL, the teams are self-managed but, the inter-team discussions are facilitated by one teacher. Within the teams, students exchange feedback on their performance in the group (Haidet et al. 2012). The main characteristics of TBL are: 

• (1) professionally relevant problems, 
• (2) small self-managed teams, 
• (3) mandatory pre-class preparation by students, 
• (4) an individual and a team test to determine students’ readiness for dealing with complex decision-based professionally relevant problems, and 
• (5) working on problems in teams (Parmelee et al. 2012). 
• Peer evaluation and feedback is another important characteristic (Haidet et al. 2012). Students are purposefully assigned to teams using a stratified random sampling process.

Similarities between PBL and TBL

두 가지 중요한 특성을 공유한다.

As learner-centered instructional approaches based on constructivist learning theory, PBL and TBL share two important characteristics: 

• (1) learning around professionally relevant problems and 
• (2) learning in small groups or teams. 

교육의 다섯 가지 원칙에 부합한다.

Both instructional approaches fit well with the first five principles of instruction of many current instructional approaches as defined by Merrill (2012): 

• 학습의 촉진에 대한 원칙 learning is promoted when (1) learners are engaged in whole or real-world problems; (2) existing knowledge is activated; 
• 학습한 내용의 활용에 대한 원칙 (3) new learning is applied and (4) integrated or transferred. In both PBL and TBL students are encouraged during group work to apply their new knowledge to problems and to integrate or transfer their new knowledge by reporting it to peers in their own words. 
• 시범사례 제시의 원칙 Furthermore, learning is promoted when (5) new knowledge is demonstrated to learners, e.g. by providing worked-out or modeling examples (Merrill, 2012). 

비록 demonstration이 PBL이나 TBL의 주요 원칙은 아니지만, 완전히 배제되는 것도 아니다. 따라서 PBL과 TBL은 Merrill이 정의한 교육의 다섯 원칙에 부합한다

Although demonstration is not a primary instructional principle in PBL and TBL as opposed to many teacher-centered instructional approaches, demonstration, e.g. in terms of lectures, is not fully neglected in both curricula. So both PBL and TBL fit well with current first principles of instructional design as defined by Merrill (2012). Despite these similarities there are differences as well (Table 2).

PBL과 TBL의 차이

Differences between PBL and TBL

촉진자 역할을 하는 교수의 수와 진행법

A first major difference is that in TBL one teacher runs various groups simultaneously (even twenty teams or more with hundred students), while in PBL there is one teacher for each group. So in PBL a teacher is physically present for each group discussion, whereas the teacher is not physically present for each team discussion. Students in PBL work in groups in different small rooms, whereas TBL teams work simultaneously in the same large room in small teams.

사전시험(iRAT, tRAT). 새로운 내용에 대해 노출되는 시점

Another difference relates to the way students attain the knowledge and insight to address the problem they are working on. TBL students are given a mandatory pre-class reading assignment while PBL students are given no preparatory reading assignment. Exposure to new content in PBL takes place after initial group discussion, mainly during self-study but also during the final group discussion. Exposure to new content in TBL takes place before team discussion.

TBL에서는 토론 전에 사전 지식을 점검함

PBL students start with an initial discussion in which they activate their prior knowledge in the group. TBL students fill out a test individually and as a group to check whether they understand the pre-assigned reading materials; so prior knowledge is checked prior to the small group discussion.

학습 문제를 누가 정해주는가

Furthermore, where PBL students identify their own issues for self-study after the initial group discussion, in TBL it is the teacher who decides based on the nature of the application problems he or she will be requiring students to solve and the conclusion of the instructional units. Further, based on the test results, the instructor decides which issues need further explanation before students work in small teams to tackle the application problems.

피드백을 받는 시점과 원리

Feedback (both confirmatory and corrective) to the students is given by peers within the group or team in both PBL and TBL. Furthermore, in both PBL and TBL the teacher gives corrective feedback to students when needed. However, in TBL students receive immediate feedback (confirmatory and corrective) from: 

• (1) fellow team members while reaching consensus during the team discussion, 
• (2) on the correctness of each team decision on the team test (via an scratch-off answer sheet), and 
• (3) from members of other teams during inter-team discussions of the application problems. 

Testing and inter-group discussion are not part of regular PBL tutorials.

동료평가

Peer evaluation and peer feedback are another difference between the approaches. Although peer feedback is also implemented in PBL in some schools, peer feedback is a structural component of TBL, with each team member having to give feedback to each of the other team members on their contributions to group learning. We will elaborate on this difference in the next section.

추론을 유도하는 기전

Learning in both PBL and TBL centers on reasoning around problems relevant to future professional practice. However, in TBL the mechanism through which the reasoning occurs is by requiring students to make questions and defend decisions to questions. TBL teams simultaneously reveal their answers in a plenary session and the teams discuss the decisions of the other teams and defend their own decision, facilitated by a teacher (Parmelee et al., 2012). Thus, students in TBL work on problems with associated questions, whereas students in PBL work on problems with no specified questions. Students in PBL discuss problems and generate their own issues or questions that need further self-study; so questions are not given.

In summary, the overall similarities between PBL and TBL relate to the use of professionally relevant problems and small group learning that both fit well with current instructional design principles, while the main difference relates to one teacher facilitating interactions between multiple self-managed teams in TBL, whereas each small group in PBL is facilitated by one teacher. Further differences are related to mandatory pre-reading assignments in TBL, testing of prior knowledge in TBL and activating prior knowledge in PBL, teacher-initiated clarifying of concepts that students struggled with in TBL versus students-generated issues that need further study in PBL, inter-team discussions in TBL and structured feedback and problems with related questions in TBL.

PBL과 TBL의 강점

Strengths of PBL and TBL

다양한 강점을 공유한다. 그럼 서로의 장점으로부터 배울 점은 무엇일까?

PBL and TBL share various strong points. They both emphasize the importance of learning around professionally relevant problems in small groups. Both instructional approaches fit well with the five first principles of instruction of many current instructional approaches as defined by Merrill (2012) and explained above. The question is can PBL and TBL benefit from each other’s unique strengths and, if the answer is affirmative, how do we achieve this?

어떤 점을 서로 제공해야 할 것인가?

What do the two approaches have to offer each other?

In addition to shared strengths, TBL and PBL have unique strengths. What can the two approaches offer each other or, to put it differently, how can PBL and TBL achieve mutual benefit by sharing strong points? How can the two approaches be combined?

PBL이 TBL로부터 배울 점은? 

How can PBL profit from TBL?

An important characteristic of TBL that might transfer to PBL is structured peer evaluation and feedback. In TBL, team members conduct mutual peer evaluation of their contributions to the success of the group and their own learning (Parmelee et al. 2012). 

Peer feedback의 긍정적 측면 

• Peer feedback has been shown to have positive effects on students’ contributions to the team and on students’ commitment to group work (Kamp et al. 2013). 
• Also, it enhances awareness of desired behaviors in the group and positive social interdependence, and it increases intrinsic motivation, mutual support and collaboration among group members (Hattie & Timperley 2007; Johnson et al. 2007). 
• Borges et al. (2012) showed that, combined with peer feedback during a traditional clerkship, TBL enhanced students’ awareness of their own emotions and recognition of the emotions of others, thereby promoting empathy. 
• Peer feedback is also a helpful strategy for promoting individual member accountability. 

In short, peer feedback is a unique strength of TBL that could prove beneficial to PBL curricula in which peer feedback is not yet implemented as a routine and structured activity.

학습목표에 대한 전체토론 도입

Another aspect of TBL that could be used in PBL is that the initial small group discussions could take place in a large class setting with one facilitator instead of a facilitator for each small group. The small groups discuss a professionally relevant problem and generate issues that need further self-study. The leader of each small group lists the learning issues generated and brings it to the teacher. Thereafter the teams report their learning issues in a plenary session. The teacher subsequently moderates a discussion in which teams challenge the choices of other teams and defend their own choices. In the end students have a list of learning issues for self-study. 

장점

An advantage of this approach is that one teacher is needed for the discussion preceding the generation of learning issues. Another advantage is that the prior-knowledge base is expanded to an entire class consisting of several small teams and that inter-team discussions are facilitated.

TBL이 PBL로부터 배울 점은? 

How can TBL profit from PBL?

An important characteristic of PBL which may be of benefit to TBL is the small group discussion before self-study in which students activate their prior knowledge, a strategy that has been shown to have positive cognitive effects on learning (Dolmans & Schmidt 2006; Van Blankenstein et al. 2011). In other words, although TBL does encourage the activation of prior knowledge by means of pre-reading assignments and does test students’ prior knowledge by asking students to answer test items, either individually or in the team, TBL could benefit from encouraging students to activate their prior knowledge by actively contributing to the group discussion before pre-reading assignments.

Another characteristic of PBL that might be of benefit to TBL is encouragement of self-directed learning. In PBL, during small group discussions, students are encouraged to generate their own questions or learning issues for self-study, which they report on and discuss in the subsequent group discussion. In other words, students plan and monitor their own learning. Encouraging students to generate their own learning issues is assumed to have positive effects from a motivational perspective and on top of that may enhance the development of lifelong learning skills. So, TBL can encourage students to generate learning issues by themselves.

The proposed exchange of characteristics between PBL and TBL as discussed is summarized in Table 3.

Conclusions and discussion

하이브리드 접근법이 가능할 수 있다.

(...)

몇 가지 추가 논의점

Some issues call for further discussion. 

• TBL은 경제적이고 자원이 많이 들지 않는다는 방식이라는 의견이 있다. PBL은 튜터가 더 많이 필요하다. 그러나 정말 TBL이 비용이 덜 드는가?
  Firstly, TBL is often claimed to be an economical or resource friendly instructional approach because one instructor can oversee the work of and ensure timely feedback to as many as twenty or more teams (Hrynchak & Batty 2012; Parmelee et al. 2012). In times of resource shortages and increasing student numbers, this claim may have been instrumental in the breakthrough of TBL. PBL requires sufficient numbers of well-prepared tutors; one for each group which requires a lot of resources (Taylor & Miflin 2008). But is TBL really less expensive than PBL? Further research will have to bring clarity on this issue.
• PBL에 TBL의 특성을 적용하는 것이 정말 가능할까? 한 명의 교수자가 동시에 여러 PBL 그룹을 운영할 수 있을까? 어느 정도까지 줄일 수 있을까? 학생선생님은 가능할까? 
  A second discussion point is the transferability to PBL of the TBL characteristic of one teacher for many small groups. Can one teacher run various PBL groups simultaneously? In other words, can a group or team work well without a teacher? This would mean losing the advantage of having a teacher available to give just-in-time information or feedback during the group discussion. However, it may be worthwhile to consider the possibility of one teacher running groups simultaneously, once students have become familiar with the PBL approach; this would mean alternate group sessions with and without a teacher as described above. Further research will have to evaluate the impact of this way of reducing teacher support in PBL. Another way of cutting back on teacher support may be to employ student teachers. Although preferably practicing physicians are involved as teachers since they know what is relevant to learn (Dolmans et al. 2013), selected and motivated senior students trained to facilitate small groups can be as effective as teachers (De Rijdt et al. 2012). The question is whether senior students will be much cheaper than experienced teachers, in view of the considerable expenses for ongoing selection and training.
• 강의, 토론, 자습, 기타 활동의 적당한 수준이 어느정도인가? TBL에 비해서 PBL에 얼마나 교수의 지원이 필요한가?
  Thirdly, there is little clarity regarding the differential amounts of time devoted to lecturing, group discussion, self-study and other activities in PBL and TBL. Also, how much teacher support is given in PBL versus TBL? To what extent are students encouraged to self-direct their learning in PBL versus TBL? These issues deserve further exploration.
• PBL에 적합한 문제와 TBL에 적합한 문제가 서로 다를 것인가?
  A fourth issue that needs further research is whether PBL and TBL differ in terms of problems that are used. Both approaches emphasize the importance of reasoning around problems. In TBL problems are associated with questions, in PBL there are no specified questions? But how do both approaches differ in terms of developing critical thinking and decision-making?
• TBL이 과연 돌파구가 될 수 있을 것인가? (특히 팀워크능력의 개발 측면에서)
  The fifth issue to be discussed is whether TBL will be the next breakthrough in health care education. Developing students’ team work skills is crucial for the health care system and will become increasingly urgent as health care is facing an increase in problems of increasing complexity requiring multi-disciplinary team work. So the urgency of paying more attention to the development of teamwork skills in curricula and educational research is undeniable; both PBL and TBL can equip students with these skills.
• PBL과 TBL중 선택해야 한다면 어떤 것을 골라야 할까? 정말 골라야 하는 것은 맞을까?
  Finally, we are faced with the choice between PBL and TBL. Which should we choose? Should we choose? In this paper, we have tried to argue that it is neither necessary nor desirable to make this choice. It may be more profitable to optimize student learning if we look for ways to combine the best of both worlds: PBL with structured peer feedback, PBL with study teams, TBL with initial group discussion before pre-reading assignment or testing or TBL with students generating their own learning issues. Choose an instructional approach that fits well with current design principles, such as the five first principles of Merrill (2012) emphasizing the importance of learning around problems, activation of prior knowledge, demonstration of new knowledge, and application and integration of knowledge. Ensure variability in problems to be discussed, order problems from simple to complex and gradually reduce teacher support (Van Merriënboer & Kirschner 2013). Start with TBL, thereafter alternate TBL and PBL to decrease teacher support, and subsequently use PBL. Start with explanation problems and gradually add decision or strategic problems. Start with paper problems; thereafter use real patient problems so as to augment the fidelity of problems. There is a range of options to include real patient problems both in the pre-clinical and clinical phase (Harden et al. 2000; Diemers et al. 2008). Combine the strengths of PBL and TBL. In sum, think win-win when designing curricula in order to optimize student learning and use varied instructional approaches that fit well with current instructional design principles.

Med Teach. 2015 Apr;37(4):354-9. doi: 10.3109/0142159X.2014.948828. Epub 2014 Aug 26.

Should we choose between problem-based learning and team-based learning? No, combine the best of both worlds!

TBL: 실용 가이드 (AMEE Guide No. 65)

Team-based learning: A practical guide: AMEE Guide No. 65

DEAN PARMELEE1, LARRY K. MICHAELSEN2, SANDY COOK3 & PATRICIA D. HUDES1

1Wright State University, USA, 2University of Central Missouri, USA, 3Duke-NUS Graduate Medical School, Singapore

Introduction

What is team-based learning?

다양한 규모에서 활용가능하다.

Team-based learning™ (TBL) is an active learning and small group instructional strategy that provides students with opportunities to apply conceptual knowledge through a sequence of activities that includes individual work, teamwork and immediate feedback. It is used with large classes (>100 students) or smaller ones (<25 students), incorporating multiple small groups of 5–7 students each, in a single classroom. TBL is specifically characterized by three key components:

다음과 같은 특징이 있다.

• 학생이 미리 준비를 해와서 individual advance student preparation;
• iRAT과 tRAT을 수행하고 individual and team readiness assurance tests (tRATs); and
• 대부분의 수업시간은 팀 단위의 의사결정 기반 적용과제에 할애된다. the majority of in-class time devoted to decision-based application assignments done in teams.

학습자 중심, 동료평가, 즉각적 피드백, 학습에 대한 팀 단위 책임이 강조됨. 한 명의 내용전문가가 20개 이상의 팀을 운영할 수도 있음.

TBL is highly learner-centered (yet has critical faculty input) and uses grading, peer evaluation and immediate feedback to ensure individual and team accountability to promote learning and, unlike other group-based instructional approaches, one content-expert instructor can instruct 20 or more teams.

여러 국가와 분야에서 활용중

TBL is used in over 60 US and international health science professional schools, including medicine, dentistry, veterinary medicine, nursing, and allied health disciplines, at several levels of training: undergraduate, postgraduate, and continuing professional education.

제대로 활용되기만 하면 학업적 성과에 대해서는 이견이 없음.

When TBL is conducted correctly, there is little question that academic outcomes are equivalent or improved in comparison to either lecture-based formats or more traditional small group learning models (McKiernan 2003; Levine et al. 2004; Koles et al. 2005, 2010; Shellenberger et al. 2009; Zgheib et al. 2010; Thomas & Bowen 2011).

다른 그룹학습과 달리 잘 하는 학생이 고통받지 않는다(혼자 다 해야하거나, 점수를 깎이거나). 모든 사람이 그 과정에 대한 책임이 있으며, 개개인이 팀에 대해 기여하는 바가 있다. 팀이 잘 협동할수록 팀과 개인의 점수가 올라간다. 팀 내에서 동료간 학습이 이뤄진다.

Unlike typical group learning, the high performers do not suffer – by either having to do all the work or poor performers dragging their scores down. The process holds everyone accountable for their own individual work and the individual's contribution to their team. The better a team works together, the better their team and individual scores. Extensive peer teaching occurs within each team.

팀 단위 점수가 잘 못하는 학생이 누구인지 가려버리는 현상을 걱정하지만, 실제로는 TBL이 개개인의 약점에 대해 더 많은 정보를 제공하고 팀 구성원과 교수가 총괄평가 이전에 그 부족한 부분에 더 많은 도움을 줄 수 있다. 또한 개인의 역량이 과소평가되지 않는다. 투명하고 명백한 절차를 밟으며, 최종 성적은 개인과 팀의 수행능력이 합해져서 나온다.

Faculty may fear that the team scores mask the underperforming student. In reality, TBL provides more data, earlier, about an individual's weaknesses and permits team members and faculty to provide help long before a summative exam. In addition, it is not as though individual performance is mitigated – it is transparent and visible – and the final grade for a student is derived from both the individual's and the team's performance.

Why the need for this Guide?

The use of TBL in health professions education is rapidly growing for at least four key reasons: 

• 대학 집행부의 요구 One is that administrators are pushing for classes to be larger (more revenue) but want them to be taught in ways that are active, engaging, and promote positive learning outcomes.
• 인증기관의 요구 Two other reasons are that accrediting bodies are also requiring documentation that schools are: (a) employing “active learning” (Liaison Committee on Medical Education 2011) and (b) equipping students with the skills they will need to work in team-oriented environments (Interprofessional Education Collaborative Expert Panel 2011).
• 학생들은 점차 수업에 잘 들어오지 않는데, 동시에 그들이 주는 메시지는 대단히 혼란스러워서 "우리에게 시험에 무슨 문제가 나올지 하나하나 가르쳐주세요"라고 하면서 동시에 "우리는 온라인으로 배울 수 있으니 강의를 가지고 우리를 귀찮게 하지 마세요. 교수님이 우리를 잘 이끌어서 생각하고 문제해결하는 능력을 키울 수 있게 해주세요"
  Finally, faculty are frustrated that fewer and fewer students attend their lectures (especially in programs where the lectures are recorded) and students give “mixed messages” about how they want to learn: “spoon feed us with detailed lectures and notes for what is on the exams” and/or “don’t bother to lecture to us what we can learn online – bring us to class when you can guide and challenge us to think and solve problems.”

What is the purpose of this Guide?

How was TBL developed?

Michaelsen when a professor of Business at the University of Oklahoma, developed the TBL strategy in response to increasing class sizes and his discomfort with lecturing and not knowing if, what, or how his students were thinking during his presentations. In addition, he feared that if students did not have regular opportunities in class to struggle with the kinds of problems they would face in the business world, the classes would be a waste of time.

How has TBL evolved?

• During the 1990s, in the USA, TBL became known and practiced in undergraduate (college-level) business schools and within other disciples in undergraduate settings. 
• In 2001, the US Department of Education Fund for the Improvement of Postsecondary Education provided funding to the Baylor Medical College in Texas to promote TBL in health professions education through faculty development workshops, symposia and the scholarship of teaching and learning. This grant spawned the adoption of TBL at many US and international medical, nursing, veterinary, dentistry, and allied health schools over the next several years, though the amount of its use at each institution varies considerably (Thompson et al. 2007a, b). The grant also supported the creation of the TBL Collaborative – a broad-based, mostly higher education, consortium that has a resource-rich website (www.teambasedlearning.org), sponsors an annual meeting with international attendance, qualifies faculty members to conduct workshops in TBL, and promotes scholarship on its efficacy, best-practices, and innovations.

필수요소 What are the essential components of TBL?

학습자의 경험은? 

Part 1: What does the learner experience?

TBL의 각 단계는 '전향적 사고'를 하게 하는데, 학생들은 '현재'를 넘어서서 '그 다음은 무엇일까?'를 생각하는 능력 - 전향적 사고 - 를 기르게 된다.

TBL's sequence of steps is Forward Thinking; guiding students into thinking progressively and gaining the ability to look beyond the “now” and constantly asking, “what's next?”

TBL sequences the learning process for the students through the following steps, as visualized in Figure 1.

Students’ perspective

TBL recurring steps

Step 1 – Advance assignment

Out-of-class/individual. Students receive a list of learning activities, accompanied by a set of learning goals. Students study materials in preparation for the TBL session. Learning activities may include readings, videos, labs, tutorials, lectures, etc.

Step 2 – Individual readiness assurance test

In-class/individual. Each individual student completes a set (10–20) of multiple-choice questions (MCQs) that focus on the concepts they need to master in order to be able to solve the Team Application (tAPP) problems.

Step 3 – Team readiness assurance test

In-class/team. This is the same set of questions that each student has answered individually! But, now the team must answer them through a consensus-building discussion. There must be a mechanism so that the team knows as-immediately-as-possible whether or not they have selected the correct answers because they need this immediate feedback to help them improve their decision-making process.

Step 4 – Instructor clarification review

In-class/instructor. Students are given clarification from the instructor on the concepts they have been struggling with during the tRAT. At the end of the Clarification Review, students should feel confident that they are adequately prepared to solve more complex problems for the next TBL step: the Team Application.

Step 5 – tAPP – Team application (가장 중요한 단계)

In-class/team. This is the most important step! Students, in teams, are presented with a scenario/vignette that is similar to the type of problem that they will be grappling with in their careers. They are challenged to make interpretations, calculations, predictions, analyses, synthesis of given information and make a specific choice from a range of options, post their choice when other teams post theirs, then explain or defend their choice to the class if asked to do so.

The tAPP's structure follows the 4 S's principles:

• Significant problem. Students solve problems that are as realistic as possible. Problems must authentically represent the type of problem that the students are about to face in the workplace or are foundational to the next level of study. The answers must not be able to be found in any source (internet, textbook), but can only be discerned through in-depth discussion, debate, dialogue within a team.
• Same problem. Every team works on the same problem at the same time. Ideally, different teams will select different options for answers.
• Specific choice. 절대로 학생들에게 긴 문서를 만들어내도록 시켜서는 안된다. 다른 팀도 쉽게 이해할 수 있도록 팀의 결정사항을 제시해야 한다. Each team must make a specific choice through their intra-team discussion. They should never be asked to produce a lengthy document. Teams should be able to display their choice easily so that all teams can see it.
• Simultaneous report. When it is time for teams to display their specific choices to a particular question, they do so at the same time. This way, everyone gets immediate feedback on where they might stand in the posting and they are then accountable to explain and defend their decision.

Step 6 – Appeal

Out-of-class/team. A team may request that the instructor consider an alternative answer to the one designated as “best.” 문제의 기술 방식을 달리 할 수도 있고, '최선의' 답가지와 비교하여 왜 자신들의 선택도 마찬가지로 최선이 될 수 있는가를 주장할 수 있다. Appeal을 한 팀 단위로 점수를 받는다. The team must either provide a clear and usable re-write of the question if they think it was poorly worded, or a rationale with references as to why their choice was as good as the “best” chosen by the instructor. Only a team that takes the steps to write an Appeal is eligible to receive credit for a particular question.

TBL non-recurring steps

Orientation

Out-of-class/in-class/individual/team. Students read a brief article about TBL, out-of-class, in preparation for the orientation session, or the course syllabus as the first Advance Assignment. In-class, students take an individual readiness assurance test (iRAT) individually, followed by a tRAT in teams and then the tAPP that covers the essential principles of TBL. The instructor clarifies TBL concepts, including how TBL is different from students’ previous learning group experiences.

Peer evaluation

Out-of-class/individual. Each student must evaluate each of his/her teammates on their contributions to the team's success and their own learning. It is best if there is both a quantitative and a qualitative component in which they get practice with framing constructive feedback to one another. It should be done anonymously, but team members are encouraged to speak directly to one another in providing feedback.

Part 2: What does the instructor have to do?

The instructor must create a TBL module in the reverse order, using a process called Backward Design (Wiggins & McTight 1998): a three-stage design process that delays the planning of teaching and learning activities until clear and meaningful learning goals have been defined and feedback and assessment activities designed (Figure 2).

Instructor's perspective

TBL recurring steps

Step 1 – Situational factors and learning goals

학생의 사전 지식을 파악한다. 구체적이고 의미있는 학습목표를 작성한다. "이 세션이 끝나고 학생이 무엇을 할 수 있기를 바라는가?" 

Identify important Situational Factors, e.g., students’ prior knowledge. Then, write clear, specific and meaningful Learning Goals that answer the question “What do I want my students to be able to do at the end of the session that they could not do before?” Be specific with exactly how well you want them to master this – use action verbs such as identify, list, explain, calculate, compare, analyze, etc.

Step 2 – tAPP – Team application

After you have established learning goals for your TBL session, you need to create or find a problem case or scenario that is authentic and believable, the kind of brief story that your learners can relate to and know that this is the sort of situation that they will soon encounter in their profession.

이러한 사례를 찾다보면 해석이 필요한 다양한 중요한 정보가 같이 딸려오겠지만, 학생들이 압도당할 정도로 많은 정보를 제공하지는 말라.

In the health professions, such cases often come with important data that need interpretation in the context of the case. Include enough to enable them, but do not give them so much that they end up being overwhelmed. You want the learners to be able to evaluate and analyze the manifest features of your problem with the data and make decisions about the questions you pose.

책이나 인터넷에서 찾을 수 있는 질문은 하지 말라.

Never ask them a question for which the answer is in a book or can be searched and found online (students are excellent web-searchers). The solution needs to be one that they can only get to the answer through their deliberations. Of course, it is perfectly OK to include elements in the presentation or data that they do not fully understand, and they must search their resources to master.

4S원칙을 따르라.

We recommend sticking with the 4 S's principles:

• Significant problem. The problem you select and the associated question(s) must be important, authentic, and truly representative of the kind of problem students are about to encounter in their professional activities.
• 질문의 질을 더 중요하게 생각하라. 팀 적용 단계에서 너무 많이 물어보려고 하는 경향이 있다. 너무 많은 내용을 커버하려고 하지 말고, TBL의 절차를 믿으라. 
  Go for quality of questions and not quantity. There will be the tendency to try to ask too much in the Team Application, fearing that you need to “cover” so much content. Trust the process: if you design questions that really make students think and struggle with making a decision about something significant; they will master the content and key concepts.

• Same problem. All teams must be working, in class, on the same problem at the same time.
• 서로 다른 문제를 줄 경우의 문제점
  Commonly, with in-class, small group exercises, each group is given a different problem with the expectation of a sharing process at the end. This is a “killer” experience for the students for two reasons. 
• 내 발표가 끝나면 다른 사람 발표에 관심이 없다. One is because they have to endure listening to other groups present their “findings,” and once they present their own, they will not pay attention to anyone else's. 
• 서로 다른 문제를 다루면 팀 간 책임을 없애버린다. 기껏해야 예의바른 질문 한두개를 할 뿐이며, 학생들은 질문을 할 만큼 자신이 충분한 정보를 가지고 있지 않다고 생각한다. The other is that, having teams on different problems largely eliminates inter-team accountability. At best, you are likely to get a polite question or two because their motivation is low and, in addition, students are likely to feel they do not have the information they need to mount a credible challenge.

• Specific choice. 최선의 정답을 선택하고, 그것을 방어할 수 있어야 한다. 이 지점에서 딥러닝이 이루어진다. Craft questions that truly probe the “why” of a concept or use a set of data for interpretation – this separates excellent teaching from mediocre. When the learner must discern between several equally plausible options, select the BEST one and prepare to defend that decision, there is deep learning.
• 팀원이 자신의 팀의 선택에 대해서 열정적으로 방어할 때가(맞았든 틀렸든) 바로 학습이 이뤄지는 순간이다.
  With TBL, by listening in to how students are processing your questions and determining the BEST specific choice, you know how they are thinking. When a team member passionately defends her team's selection against countervailing positions, whether she is wrong or right, you know that you have a “teaching moment.”
• 어떤 교수들은 강의를 마치고 나서 박수갈채를 받으면 스스로 훌륭한 선생님이 된 것처럼 느끼지만, 그 50분 동안 어떤 순간에 학생들이 실제로 당신이 가르치는 내용에 대해 생각을 하고 있는지를 정말 알 수 있는가? 새로운 상황이나 문제에 배운 내용을 적용할 수 있을지 여부는 언제 알게 되나? 시험볼 때?
  Some instructors feel they are excellent teachers when students applaud after a lecture. But, at what point during the 50 min do you really know how students are thinking about the concept you are teaching? When do you know if they can apply that concept to a novel situation/problem? At the exam?

• Simultaneous report. A key to energizing team discussions is using procedures that make teams accountable for reaching and being prepared to defend a decision. Having teams work on the same problem is essential for intra-team accountability but it is not enough.
• 보통 교수자가 책임을 회피하는 두 가지 방법이 있다 : (1)자원자 찾기(다른 조가 리스크를 떠안기를 마냥 기다린다) (2)무작위로 부르기 (처음이라 운이 나빴네요 or 앞에 한 팀의 의견에 동의합니다.
  Instructors often reduce accountability in one of two ways. First, students discover that you have a pattern of calling for volunteers (e.g., “which team would like to give their answer?”). Students know that it is pretty safe to sit back and let the other teams risk giving what might be an incorrect answer. Second, if your practice is to randomly call on one of the teams to give their answer, students’ motivation to “get it right” is significantly reduced by their realization that, even if they do not have an answer, they have two minimally embarrassing options. If they are the “unlucky” team that gets called on first, they can say, “We haven’t had time to reach an agreement and, if they aren’t the first team to be called on, they can say, ‘We agree with team ___’ (i.e., the team that reported first).”
• 그러나 TBL에서는 모든 조가 동시에 자신들의 결정을 밝히고 그것에 대한 대응논리를 만들기 때문에 위와 같이 되지 않는다. 
  With TBL, however, teams are fully accountable because students are informed from the beginning that all of the teams will have to report their answers at the same exact moment and their task is to make a decision and be prepared to defend it. As a result, the interaction during the Team Application follows a distinct pattern. When the learners first begin grappling with the problem, there is often quiet in the room as they read and ponder individually for a few moments, then a low-grade “buzz” starts as members of teams start sharing their impressions, raise questions, and assign searching tasks to one another – in short, enjoin a strategy for making a decision within the timeline of the exercise.
• 자기 조의 최종 결론을 밝혀야 되는 순간이 오면, 모든 팀은 다른 사람이 모두 볼 수 있도록 동시에 답을 내놓는다. 
  When time is called for the posting of all decisions, teams simultaneously post their answers for all to see (e.g., by displaying a numbered or lettered card, putting up a poster, using Audience Response System “clickers,” etc.).
• 팀 내의 논의가 이뤄지는 동안 교수자는 방을 돌아다니면서 어떤 대화가 이뤄지는지 보고, 들은 내용을 기억했다가 이어지는 토론에서 활용하는 거시다. 다른 소그룹과 다르게 각 팀은 같은 문제에 대한 답을 내놓아야 하고, 다른 팀과 경쟁을 통해서 상당한 논의과 활력이 오간다. 
  If the Team Application case and questions have been written well and are tightly linked to the learning goals and the readiness assurance process (RAP), then, it is rare that a team will have questions for the instructor during the time they are trying to reach a decision. As the instructor, your task is to roam around the room and listen in on the team conversations and learn how they are processing the assignment, remember what you hear and use it in the class discussions to follow. Unlike any other small group work, the requirement that each team must make a decision on the same difficult question, and be in competition with other teams, generates a great deal of noise and animation in the classroom. The better your questions, the more lively the room becomes until the time for posting decisions.
• 각 팀의 결정이 공개되면, 교수자의 임무는 각 팀의 최종결벙을 활용해서 팀간 토론을 유도하는 것이다. 
  Once the teams’ decisions have been revealed, your job is to use the teams’ simultaneous reports as a catalyst for facilitating the inter-team discussions. What are the ways to facilitate the class discussion, generating dialogue and debate between teams? Go for the “Why?”: “Why did your team make this decision?”, “Explain your thought processes.” To a different team with a different answer: “Let's hear your rationale, why is it better than what we have just heard?” If there are many teams in the classroom, it is not necessary to get an oral defense from each team – this may bore the others and be too much like the usual small group work project sharing.

촉진기법 팁

Some facilitation tips:

• 유용한 어플
  When you call on a team to explain, select a specific student from a random list sheet or use Teacher's Pick™, an APP for the iPHONE or iPAD (http://itunes.apple.com/us/app/teachers-pick/id320221052?mt=8) – do not ask for a representative of the team to speak because the team's extravert will always do it.
• 말하는 학생은 일어나서 하도록
  Make a rule that anyone who speaks to the whole class must stand up or use a microphone, and get in the habit of moving away from the student who is speaking so that he/she will speak louder.
• 교수가 아니라 팀원한테 말하게 하라. 교수자의 결론은 최대한 아껴두라. TBL에서 학습은 교수자가 던진 질문에 의해서 프레이밍된다. 
  At first, the speaking student will try to talk to you since you asked the question. Inspire them to speak to the other teams, not you. Remember: your teaching moments are framed by the questions you ask about their decisions, their thought processes; save your own conclusions about the question until they are all in suspense about the “best” decision, then explain using what you have heard in their propositions.
• 체크리스트
  Before the session, review each of your readiness assurance test (RAT) and Team Application questions with this checklist:
• What is the key learning point from this question?
• Where would the student have been exposed to the information needed to answer it?
• What if all teams get it right? Do I move on to the next question? Do I give them a few more minutes to post what they think is the “Second Best Answer”?
• What if all teams get it wrong? How will I show them my thinking about what I selected as the right answer?

Step 3 – Individual readiness assurance test/team readiness assurance test

국가시험 수준의 우수한 MCQ가 필요함

Preparing the Team Application (tAPP) first, enables you to design RATs that truly prepares the learners for the tAPP – you know what they need to know to apply the content to meaningful problems. The questions should be in multiple-choice format (MCQ) and they should be well constructed so that their quality is equivalent to your end of course/term or licensing examination. A great source for writing effective MCQs is the National Board of Medical Examiners Item Writing Manual, downloadable at their website (www.nbme.org/publications/item-writing-manual.html).

몇 개 문항이 필요할지는 내용의 복잡성 정도와 가용 시간에 달려 있다.

How many questions you prepare depends on the amount and complexity of the content, and how much time you have in your course design for TBL. On the one hand, you do not want to overwhelm the learners with lots of MCQs where they feel they are always studying to take a test, but you and they do want to have regular assurance that they are mastering the content and that the work they have done preparing is important by being assessed.

문항은 큰 그림에 초점을 둬야 하며 세세한 내용에 대한 것이어서는 안된다.

Another key is that the questions should focus on the big ideas not the details. If they really understand the big ideas, they are prepared to learn the details when they try to use them to make the decisions that are part of the Team Application.

There are two parts to the RAT: individual and team. Learners take the iRAT at the beginning of the session, recording their decisions using a Scantron™ or Audience Response System “clickers.”

IF/AT 활용

When time is up, teams cluster and answer the exact same questions as a team, with the tRAT, making their selections on an Immediate Feedback Assessment Technique (IF-AT™) form. The IF-AT is a multiple-choice answer form with a thin opaque film covering the answer options. Instead of using a pencil to fill in a circle, students scratch off the answer as if scratching a lottery ticket. If the answer is correct, a star appears somewhere within the rectangle indicating the correct answer. Students earn partial credit for a second attempt and learn the correct response for each question while taking the tRAT. One member of a team is picked by the team to do the scratch off on the IF-AT form, and all are at rapt attention as he/she determines whether or not the team's decision on a question is the preferred one. Generally, teams will give out a small cheer when right and a light groan if wrong. If they do not get it right the first time, they will immediately re-engage on that question and make another selection, but not without careful consideration since the stakes are higher. More information about the IF-AT form is available at the Epstein Educational Enterprises website (www.epsteineducation.com).

Always structure time for the discussion of the RAT after the team process, and when you use the IF-AT, encourage teams to select the one or two questions that they would like the whole class to discuss. Although they know your designated “correct” answer from the IF-AT, if you crafted the questions well, then there should always be two to three questions that really challenged them and they will want to discuss, or even appeal (see step 6 – Appeal).

왜 IF-AT를 사용하기를 권고하는가?

We strongly recommend using the IF-AT in the tRAT. These are our reasons:

• tRTA은 각자 학습해 온 것을 설명하고 공유하게 만든다. 처음에는 투표로 하다가 나중에는 토론을 통한 합의를 이뤄나갈 것이다.
  Whether it is the very first gathering of the team or the last in a course, the tRAT forces them to share what they have learned as they each explain why they support one answer over another. Because conflict is uncomfortable, brand-new teams will initially make their choices by voting. However, with time (and the IF-AT form speeds up the learning process) they learn that voting is risky and it is more effective to share explanations first, then discuss their way to a consensus.
• 팀 응집력을 강화시켜준다. 스크래치를 긁을 때 모든 사람이 모여들 것이다.
  The IF-AT promotes team cohesion – when the team makes its decision and scratches off the selected option on the IF-AT, everyone is paying acute attention: if they have drifted apart physically during the discussion, they move back in; if they are separated by a table, they lean in – everyone wants to see and they will need to get closer to do so.
• 오답일 경우, 부분점수를 받기 위해서라도 정답을 찾는 작업에 바로 착수하게 된다.
  If a team “gets it wrong” the first time, they immediately explore why so that they can select the “right” one next, and they are motivated to do so since they will still get partial credit.
• 지나치게 주장이 강한 사람도 언젠가 틀린 답을 낼 수밖에 없고, 이 경우 그 학생은 조금 더 조심하게 될 것이다. 
  Learners who tend to be overly assertive will inevitably be wrong on one or more of the questions. When this happens, the pushy student will become more cautious and the peers will be more willing to advocate for further considerations of an answer. Or, a learner who usually “goes along” with the crowd will eventually be put on the spot and asked to defend their choice – in effect, be encouraged to participate, especially if they have been correct and not helped the team get to the right answer.

Step 4 – Advance assignment

사전학습자료는 교수자게 제공하는 Scaffolding 같은 것이다.

Prior to coming to class, the learner needs to know what he/she must read, watch or do to be prepared for class. This is where you, the instructor, provide “scaffolding” for their acquisition of the information (content). It works best to provide them, as far in advance as possible, what must be read or done and clearly identify the level of requisite mastery so that they can be successful. You should develop or select appropriate teaching and learning activities (readings, videos, labs, tutorials, lectures, etc.) for the Advance Assignment that are aligned with the iRAT/tRAT questions, are effective and sufficient for content coverage, and that include specific learning goals.

의학의 세부사항은 무한하기에 모든 내용을 주고 싶겠지만, 핵심 개념만을 찾게 해주고, 그 개념이 무엇이며 어떻게 적용하게 되는가를 알려주는 것이 좋다.

In the health sciences, there is an infinite amount of detail within many critical complex concepts. As tempting as it is to list all the content that they must learn and then test them on it in the RATs, it is far better to identify the key concepts and inform students what the concepts are and how they will be expected to apply them. For instance, autonomics is one of the most complex areas in medicine, and there are a great many agents whose names must be memorized along with how they affect different receptors. Your “scaffolding” lecture or tutorial clarifies the principles of autonomics, reviews the body's anatomic structures for sympathetic and parasympathetic systems, notes a few of the prototypical agents/transmitters, and leaves them with a handful of practice problems; the answers to be provided the next day or posted online. Memorizing the many agents that interact with the autonomic nervous system is a requisite task, like memorizing the multiplication tables, and doing so will “stick” better if done in the context of the “bigger picture” with as many practice exercises as possible.

어떻게 준비할지는 학습자가 결정한다.

Learners decide how they can best prepare: some study alone, some will form a study group, and some will use their class team as a study group. You do not have to recommend or suggest any; let them figure out what works for them as individuals and as teams.

After the first couple of TBL sessions, students will have determined how much time and energy they need to devote to the Advanced Assignment. Their scores on the iRAT, tRAT, and tAPP let them know how they are doing with the material as the course moves along.

Step 5 – Instructor clarification review

교수자가 RAT에서 어려웠던 개념을 명확히 해주는 단계. 그러나 이 부분은 강의도 아니며, 모든 개념을 리뷰해주는 것도 아니다. 

The RAP should include an instructor Clarification Review, in which students get clarification from instructor on the concepts they have been struggling with during the tRAT. At the end of the Clarification Review, students should feel confident that they are adequately prepared to solve more complex problems for the next TBL step: the Team Application. An effective Clarification Review predicts/addresses knowledge gaps (focused on the concepts that the students are struggling to understand), is neither a lecture nor a review of all concepts, and supports the development of critical thinking skills.

You should design the Clarification Review only after you have created the tAPP and RAT questions, since the purpose of the review is to better prepare students for the tAPP by addressing learning gaps identified during the tRAT.

Step 6 – Appeal

어필을 받아들이게 되면, 그 팀은 추가점수를 받게 된다. 두 가지 이유가 있다.

Teams should be able to Appeal a question in the RAP or the tAPP. If you accept their Appeal, then only that team is awarded the credit. There are two reasons for an Appeal:

• the team thinks that they were misled by the way the question was written and, to have the appeal granted, they must re-write it so that you agree that it is much clearer;
• the team is convinced that their answer is best and they can support this position through argument and/or valid source material which they reference.

어필이 들어왔을 때 생각할 시간이 필요할 수 있다.

Sometimes in the class discussion you hear an argument for an answer that is compelling, one you have not even thought about and, even though it is not normally done, you can award credit on the spot for their brilliance. More often, you need some time to process the arguments, as do the teams submitting appeals.

모든 학생들이 어필에 대한 당신의 결정을 알게 하라

Keep the turn-around time as brief as possible and let the whole class know of your decisions. If the Appeal process is done in the spirit that everyone, including the instructor, can learn more, then it encourages more discussion and deeper learning.

자신의 주장을 무한정 내세우려는 일부 학생들에 의해서 곁길로 새지 않아도 된다는 장점이 있다.

A distinct advantage of a genuine process for Appeals is that you will not be sidetracked during the class by a few students who want to argue their position ad infinitum. It will allow them and you some reflection time and opportunity to better articulate a position.

TBL non-recurring steps

Team formation

최대한 오래 같이 있을수 있는 팀을 구성하라

Ideally, create teams that can stay together as long as possible. Sometimes this means for a year, a semester, or even a 4-week clinical rotation. There are four principles for assigning students to teams:

• 스스로 팀을 짜게 하지 말라 Never let them self-select!
• "플러스 요인"이 무엇일지 고려하라 Determine what you consider to be “wealth factors” in the class, for instance, previous work experience in healthcare or having an advanced degree in a health science field like biochemistry or physiology. Distribute all such students across different teams.
• 최대한 각 팀의 구성을 다양하게 하라 Ensure that each team has as much diversity as possible. This is highly contextual and you must explore the potential characteristics in the class that represent diversity. For instance, if your school draws from a wide geographic area that includes rural and urban settings, the teams will benefit from having members from both. Gender balance, if possible, is also desirable.
• 팀 구성 단계를 투명하게 밝혀라 Make the assignment process transparent. Students should never wonder how they were assigned to a particular team.

Orientation

대부분의 학생은 TBL이 새로운 경험이므로 오리엔테이션이 필요하다.

For most students, TBL will be a new experience. The TBL Collaborative website has a link called “Orienting Students” with tips on how to introduce TBL to your students through a TBL sample session (www.teambasedlearning.org/Default.aspx?pageId=1032382). You can create the sample session using TBL content (based on a brief TBL article), or using the course syllabus as the first Advance Assignment.

가장 어려운 부분 중 하나는 학생들이 미리 준비해오도록 하는 것이다. 학생들은 '시험만 잘 보면 되지 왜 수업에 준비를 해와야하지?' 라는 의문을 가질 수 있다.

Often the biggest hurdle is student attitude about preparing for class – so many are accustomed to coming to class to be told what will be on the exam, so why prepare for a class? With TBL, they must prepare using the Advance Assignment in order to pass or do well in the course. Classroom time shifts from being a time to transmit information to problem-solving with course content that is learned largely outside of class. This becomes very clear during the Orientation session.

Peer evaluation

동료평가의 중요성에는 이견이 없다.

There is little question about the importance of our students in the health professions learning how to give and receive constructive feedback from peers since they will need this skill set in the work setting with team members from several disciplines. We recommend the development of a process that encourages students to highlight the positive behaviors of their peers and develop the skills for constructive feedback. There are several viable models for conducting peer evaluation, all of which include:

• 각각의 구성원을 팀에 대한 기여도와 개인별 학습으로 나누어 평가
  evaluation of each teammate on his/her contributions to the team's success and their own learning;
• 양적, 질적 요소 포함
  both a quantitative and qualitative component;
• 효과적인 피드백을 주는 가이드라인 제공
  guidelines on how to provide helpful feedback.
• For instance, for a qualitative query:
• “What is the single most valuable contribution this person makes to your team?”
• “What is the single most important thing this person could do to more effectively help your team?”

Grading

As with Team Formation, there are some principles for how to grade TBL:

• TBL이 전체 과정에서 반영되는 시간의 비율이 반영되어야 함. The percentage of time of a course that is devoted to TBL must be reflected in the course grade. For example, if it is a 12-week course and TBL is used for about one half of the contact hours, then it should count for about one-half of the final grade.
• TBL의 각 요소에 가중치를 두어서 Each component of TBL has a weight in the grading scheme and it will work best if the students have some responsibility in determining this within the limits you set.
• 동료평가도 포함되도록 The peer evaluation should also count as part of the TBL grade.

An example of a TBL grading scheme would be:

iRAT = 25%

tRAT = 35%

tAPP = 35%

Peer evaluation = 5%

What are the TBL critical contextual factors?

대학의 문화 Institutional culture

For TBL to be successful in a course or throughout a curriculum, it needs sanction if not support from the administrative leadership. Often, the simple support position of “classroom time needs to be used for solving problems and not just transmittal of information” goes a long way for faculty to consider using TBL.

Although there is good support in the literature for the effectiveness of TBL for a wide range of subject areas (www.teambasedlearning.org/refs), there are strong biases by faculty and administration against any active learning in the classroom. Some very experienced, and often talented, instructors grew up on the lecture-format and are wedded to it as the best way for students to learn – to be exposed to great minds such as theirs.

The administrative leadership may give in to these “sages on the stage” if they are large grant generating faculty whom they do not want to lose to the competition. In our experience, one faculty member or a small group of faculty can get started with TBL and generate both positive academic and student satisfaction outcomes within a couple of years. Students can become the best “salespersons” for having active learning in the classroom.

교수 개발 Faculty development

There are several steps that a group of faculty can take to make their transition to using TBL successful:

• 관련 워크숍 참여 Participate in several training workshops on TBL. If your aim is to have a few individuals experiment with TBL, have them attend the TBL Collaborative Annual meeting or equivalent venue. If your aim is to have a substantial part of your curriculum taught with TBL, you will need to bring one or more consultants to your campus first to introduce TBL to your faculty and later to help them refine their TBL course design and delivery.
• TBL학습 커뮤니티 개발 Once you have some pilot TBL courses up and running, establish a TBL learning community on campus or with neighboring institutions.
• TBL 전문가로서 조언을 받을 수 있는 조언자 찾기 Identify a consultant, someone experienced with TBL who can critique materials, observe initial sessions and help troubleshoot the problems that will inevitably arise.
• 모든 모듈 자료에 대해서 동료평가 받기 Peer-review all module materials, especially the MCQs in the RAP and the questions for the tAPP – this must include careful editing of the questions for grammar, syntax, and format, matching of module objectives to the materials, and framing of tAPP questions so that they will generate good, thoughtful discussions within and between teams.
• TBL의 교육과정 설계의 핵심이 되도록 하기 Ensure that TBL is an integral part of the course or curriculum design – it will flop if it is just plugged in without being well-linked to the other components (Fink 2003).
• 학생을 참여시키기 Involve students, both to introduce them to how TBL works and get constructive feedback from them after a session.

학생 참여 Student buy-in

학생들은 오리엔테이션을 받아야 한다. Wright State University Boonshoft School of Medicine 의 사례

Students must have orientation to TBL, and there are several ways to do this. 

At Wright State University Boonshoft School of Medicine (www.med.wright.edu), we have used the following:

• 제1일 First day of class, give them the article “Three Keys to Using Learning Groups Effectively” (Michaelsen 1998) to read in class, then form teams, give them a iRAT and tRAT on TBL from the article, end with a couple of application questions about TBL.
• 수업 전 Before class, ask students to study the course syllabus, i.e., grading, attendance, papers due, key topics, exam schedule. At class, conduct a TBL session on the syllabus. After this first TBL session, make time at the end to review with them the objectives for the session and ask if they feel they met the objectives through the TBL experience.

Duke-NUS의 사례

At Duke/NUS Singapore (www.duke-nus.edu.sg/web), they conduct a TBL session as part of the medical student interview process; highlighting the value and benefits of TBL, the general process, and use it a bit as a marketing tool. Once students are accepted, they participate in a series of workshops that reinforce the process, the skills for team development, review of how study habits might change, and a practice session with a review.

공간, 음향시설 Space/acoustics

TBL은 장소에 크게 구애받지 않는다. 

TBL is very adaptable to a variety of space restraints, such as fixed seating in an auditorium. However, the best space is one in which students can easily cluster in either a small circle or around a small table. We say small table because we have seen the use of larger, eight to ten seat tables and students cannot get close enough to really “team;” besides, the large tables will have space for laptops and lots of references which get in the way of good team discussions. Remember: your Team Application problems can only be solved through discussion and deliberation, not a web search!

교수의 이상적 위치는 교수가 모든 학생을 볼 수 있으면서 모든 학생이 서로를 볼 수 있는 곳이다. 

The ideal setting has a spot for the instructor in which he/she can see everyone in the classroom AND all students should be able to see each other. If a student can stand up, speak and be heard by all in the room, then you do not need an amplification system. The best amplification system has microphones at each table or cluster area; passing around a single mike can be cumbersome.

Why do we feel that TBL is an excellent instructional strategy for education in the health sciences?

What are the outcomes to date and what are important questions to answer in the future?

TBL in medical education is relatively new, and the evidence for its academic effectiveness is only beginning to grow (Nieder et al. 2005; Letassy et al. 2008; Shellenberger et al. 2009; Koles et al. 2010; Thomas & Bowen 2011).

변형된 형태를 사용하는 경우가 많다. 

One of the confounding issues in evaluating the literature to date on TBL is that authors have modified the strategy, sometimes extensively, and do not indicate this in the title. In fact, one must scrutinize the methods to learn exactly what was done, i.e., how were teams created, were the three key components used?

의학교육자들은 점차 TBL 모듈을 디자인하는 것에 능숙해지고 있다.

We feel that as medical educators become better at designing TBL modules, ensuring their integration in a course or curriculum, and clarifying what the desired academic outcomes are, the results will be very positive, especially in contrast to a pure lecture-based curriculum.

다음의 연구가 필요하다

We also feel that there are non-academic outcomes that are particularly important for future investigation:

• Does peer evaluation lead to enduring positive changes in how students collaborate?
• How does the decision-making process within a team help students make better decisions independently?
• Since there is emerging evidence for “collective intelligence” within small groups, (Woolley et al. 2010), what characteristics should we use to assign students to teams?
• Does TBL improve clinical reasoning and/or critical thinking skills? If it does, then how can we enhance this outcome?

Why is TBL unique in small group learning?

Small group learning, when done well, as described in the recent AMEE Guide 48 (Edmunds & Brown 2010) provides students with rich opportunities to explore, explain, and understand course material while learning how to communicate, collaborate, and problem-solve as they might in the workplace.

PBL과의 차이

Since PBL is probably the most commonly used small group learning strategy in medical education to date, we consider it relevant to highlight the several differences between PBL and TBL (Table 1).

What additional characteristics make TBL a good fit for healthcare professions education?

TBL의 장점

These are additional characteristics of TBL that make it unique and particularly well suited for health professions education:

• 다양한 활용도 Versatility of use. Large or small classes; single or series of sessions; whole or portions of a course; blends with lectures, labs, other learning activities; inter-professional education activities.
• 효과적 팀 구성 Effective team formation. Teams are created thoughtfully and transparently, and ideally teams stay together for as long as possible.
• 교실 밖 준비 Out-of-class preparation. The Advance Assignment informs the learner what he/she needs to master before coming to class in order to be prepared for the RAP and decision-based application assignments that follow.

Immediate feedback

Immediate feedback on both individual and team performance is inherent in the process so that students know “where they are” with respect to understanding both the content and its application. The instructor also has continuous opportunity during the class period to know how learners are thinking about the material as they grapple with using course concepts to solve real-world problems and make medical-practice decisions.

• Peer evaluation. Peer evaluation counts and, when the process is properly designed and managed, learners learn how to provide constructive feedback to peers and learn how to adjust their own behaviors to become more effective participants within their teams. This is an increasingly important component of TBL since healthcare professionals are frequently evaluated by their patients, colleagues, administrators, and other members of an allied healthcare team.
• Authentic problems. The primary emphasis of the instruction focuses on solving problems, usually based on case vignettes with realistic data and images, that are as authentic as possible, and the choices that the learners have to make force them to partake in the decision-making process that they will regularly encounter in the clinical arena.

Med Teach. 2012;34(5):e275-87. doi: 10.3109/0142159X.2012.651179. Epub 2012 Apr 4.

Team-based learning: a practical guide: AMEE guide no. 65.

TBL 촉진법 (Medical Teacher, 2015)

Twelve tips for facilitating team-based learning

CHARLES GULLO, TAM CAM HA & SANDY COOK

Duke NUS Graduate Medical School, Singapore

TBL의 기원: 1970년대, 대학원 경영학 과정에서 사용하였음.

Team-based learning (TBL) traces its roots to Professor Larry Michaelsen at the University of Oklahoma, United States (USA) in the late 1970s. Since then, it has grown to become a popular and effective instructional strategy used in a number of different educational settings (Koles et al. 2010; Parmelee & Michaelsen 2010a; Kamei et al. 2012; Fatmi et al. 2013). Although Michaelsen implemented it in graduate business instruction, it has more recently been used as a major teaching platform by a number of different educational programs across the United States (Team-Based Learning Collaborative 2013) and in a number of medical schools (Thompson et al. 2007a,b). At the Duke-NUS Graduate Medical School in Singapore, we have been using this teaching methodology since 2007 as a primary mode of learning for our students during their pre-clinical instruction (Kamei et al. 2012). The benefits of this teaching methodology are numerous, and have been well-documented in a number of sources (Hunt et al. 2003; Zgheib et al. 2010; Sisk 2011; Fatmi et al. 2013; Hazel et al. 2013).

TBL개요

For those who may not be familiar with the structure, in brief, students are placed in teams of 5–7 learners. These teams stay together for an extended period of time. The TBL process is made up of three phases: 

• (1) Preparatory phase, where students are given material to study before they come to class. 
• (2) Readiness assurance test (RAT), where students take both an individual (IRAT) and a team (TRAT) test to assess their understanding of the pre-class material. 
• (3) Application phase, where students apply what they have learned in meaningful case-based exercises. 

During this phase, students are asked to work in teams to apply the knowledge formally assessed during the readiness assurance phase (IRAT/TRAT; refer Figure 1 for an illustration of these steps; Michaelsen & Sweet 2008). While a seemingly simple process, the heart of creating an engaging and impactful inter-team discussion is an effective facilitation. Yet, effectively facilitating these discussions can be one of the most challenging aspects of TBL.

다른 소그룹 교육에서 촉진(facilitation)과의 차이점: 기본적인 촉진기술은 비슷함

It is important to first define how TBL facilitation may differ from facilitation in other small group settings and then where in the TBL process facilitation is typically used and critical. While basic facilitation skills are necessary in all learning situations; unlike lectures, small group or problem based learning (PBL) environments, TBL’s added challenges for facilitators is in the inter-team engagement, keeping all learners engaged and accountable, eliciting the answers from the class, and challenging learners understandings and assumptions (before revealing the faculty’s answer).

촉진이 필요한 세 가지 상황이 있다.

There are three places where directed facilitation occurs in TBL (Figure 1). 

• TRAT직후의 토론(facilitation 없는 상태에서의 잘못된 이해나 개념 바로잡아주기) The first place that a facilitated discussion occurs is after (and importantly not during) the team readiness assurance test (TRAT). This facilitated discussion requires faculty to be able to draw out the misunderstandings or resolve any misconceptions not addressed by the non-facilitated inter team TRAT discussion and reveal “correct” answers. Being skilled at guiding the students to discuss and ask questions about their uncertainties maximizes the learning.

• 적용단계 이후의 학습: 역동적 토론을 유도하고, 어려운 개념을 이해한바대로 설명하도록 하고, 자신의 생각과 다른 사람의 생각을 비판하는 것. 예상하지 못한 질문에 대응하는 것이 가장 어려움. The next place where facilitation is critical in TBL and further enhances the learning experience is after the application phase. The post-application discussion requires faculty to create a dynamic classroom discussion and assist students to articulate their understanding of difficult concepts or critique their own and others thought processes when solving the problems. It is through this inter-team discussion where a deeper understanding and learning can be achieved. In both the post-TRAT and application setting, one of the most difficult activities faculty have is in managing the unexpected questions and eliciting rather than giving answers.

• 적용 후 팀간 토론: Assimilative learning과 Transformative learning을 촉진하는 단계이므로 어렵다. 이 단계에서 필요한 촉진기법은 “Elaborative Interrogation Technique,”이라고 불림. Managing the post-TRAT and application discussions between multiple teams in a classroom setting requires a very different set of skills from managing the typical learning environment to which most faculty are accustomed. One reason for this difficulty is that the RAT and the application phases of TBL encourage both assimilative learning, the process of incorporating new information into existing knowledge structures (Seel 2012), and transformative learning, the process of altering existing knowledge structures through critical thought (Mezirow 1991). The latter is enhanced through the discussion with team-mates who learn from one another (Meers-Scott et al. 2010). Appropriate facilitation after the team’s independent discussion enhances the transformative learning process if performed effectively. This type of facilitated learning is referred to as the “Elaborative Interrogation Technique,” an effective learning methodology (Dunlosky et al. 2013). Although some of the properties of being a skilled facilitator may appear to be more of an art than a science, we believe that many of the skills can be learned. Below are our 12 tips for effective facilitation, aimed at providing faculty with skills that are useful in promoting effective learning within a team-based environment. We have divided the tips into two categories: creating the right environment and enhancing active engagement of learners.

적절한 환경 구축하기

Creating the right environment

The major role of the facilitator is to create a safe and engaging learning environment while still managing the flow and time. These first six tips focus on the process of TBL, role of facilitator, and activities a facilitator can do to best create the right learning environment.

4S를 활용한 매력적인 문항 만들기

Tip 1 Use the 4S’s to craft engaging questions

Using all aspects of the TBL structure goes a long way in ensuring full student participation. The IRAT followed by the TRAT ensures that students come well-prepared to discuss issues and identifies their gaps so they are ready to learn. Using the four S’s of application writing and development (significant problem, same problem, specific choice, and simultaneous report) can also ensure maximal participation and active engagement during the application phase (Parmelee & Michaelsen 2010b). The use of significant/authentic problems, having everyone working on the same problem, requiring them to make a single choice (and defend it), and enabling simultaneous reporting, is the start of creating a problem that will enable a more stimulating environment in which to facilitate.

시간을 확인하기

Tip 2 Watch the clock

TBL에서 토론은 전면의 중앙에 나서게 된다(front and center). 토론에 적어도 절반의 시간은 배정되어야 한다.

One of the most difficult tasks for a facilitator to perform in the TBL classroom is to keep within the allotted time. One of the benefits of TBL is that discussions are brought “front and center”. Ensuing that various arguments are brought out in the open can be time consuming. Therefore, the first priority is to make sure that enough time has been scheduled for the discussion phases. 

Ideally, the discussion phase should be at least half the time allocated. 

• In a 50-min class, that would be roughly 25 min for RATs (both individual and team) and 25 min for discussion. 
• The application phases should follow the same guidelines. A 25-min application should be accompanied by a 25-min discussion phase (longer if possible). 

The facilitator must constantly be alert for diminishing time and make adjustments as he/she goes along as to how much discussion is to be pursued. This may mean interrupting students who are taking a very long time to respond, asking faculty to cut short their explanation or tactfully interrupting those who have launched into a full-length didactic lecture, or even dramatically shortening the team-to-team discussion phases after some questions. Being aware of the time means also being realistic about the number of questions in each session. Having too many questions and insufficient time to review and discuss can be frustrating to the students.

수업 전에 촉진 단계에 대한 전략을 세우기

Tip 3 Strategize the process of facilitation with faculty before class

It is important to strategize before the session how the questions should be facilitated, especially if you have a facilitator who is different from the course director or content expert. In practice, not all questions require an in-depth facilitated discussion. Certain questions may have limited learning points which students can grasp easily by themselves and thus need not be discussed at length if time is short. One time-efficient way to implement this approach is for faculty to consider a facilitation strategy for each RAT question while students are working on the questions. 

적용질문에 대해서는 팀별로 답안을 작성하는 즉시 제출하게 해서 교수가 미리 어떤 문항의 facilitation이 필요한지 준비할 수 있게 하면 좋다. 모든 팀이 답을 다 맞췄더라도 facilitation이 필요한 문항과 그렇지 않은 문항에 대해서 미리 대비. 팀별로 답이 다 다르면 서로 토론하게 하는 것이 좋음

For the application questions, it is ideal if teams turn in responses as they complete them so faculty are informed of the answers once the students have chosen and consider the facilitation strategy based on the answers given. For example, if all teams answer a particular question correctly, the facilitator may choose not to have an in-depth facilitated discussion as there may be no further learning points to be covered. Conversely, if all teams answer the question correctly but there are critical learning points, or faculty want to be sure the teams derived the correct answer for the right reason, such a question may benefit from a facilitated discussion. The faculty may also decide that if the teams choose many different answers for the same question, a good strategy may be for the teams to debate, as a whole class, why they chose their answer.

촉진이란 내용을 전달하는 것이 아님을 기억하기

Tip 4 Remember facilitation is NOT delivery of content

촉진이란 참여하는 분위기를 만들고 학습자가 느끼는 위협을 최소화하는 것이다.

'내용 전문가'의 모습은 벗어던지고 '촉진자'의 입장이 되어야 한다.

As teachers, we want to ensure our students receive the necessary information. Thus, in TBL, it can be difficult to remember that at certain points one must be a facilitator of learning, not the deliverer of content. Carl Rogers discussed in his publication “Freedom to Learn” (Rogers 1969) that the facilitator in a classroom is one that “creates the environment for engagement” and is obligated to create an environment where “the threat to the learner is reduced to a minimum”. He was a strong proponent of minimizing one’s expertise as much as possible when facilitating in an educational setting so as to avoid teaching a person directly, but facilitate his or her own learning. This sentiment is at the heart of TBL. Thus, when facilitating in a TBL learning environment, removing the “content expert” hat and putting on a “facilitator’s hat” may be the single most important and difficult step faculty face when engaging students.

촉진 단계에서 답을 알려주는 실수를 하지 않기

Tip 5 Avoid giving away answers during facilitation phase

중립적 태도를 취해야 하며, 찬성 혹은 반대의 입장을 표명하는 순간 토론은 단절된다. 스스로의 정체성을 내용전문가가 아니라 촉진자라고 설정해야 한다. 

As the role of the faculty during the facilitation phase is to “facilitate” the learning and elicit information from the students, it is critical to remain neutral and non-judgmental with the discussion as it develops. During the learner debate phase of the discussions, any sign of approval or disapproval of a comment or response will shut down the discussion immediately. As the goal of the TBL in-class facilitated discussion is to ask certain questions to ascertain the student understanding and knowledge of concepts and to encourage them to articulate these concepts and main points of an argument and to teach one another, keeping the discussion going is paramount. It is often very difficult for a faculty member to hide their opinion when a given response is factually incorrect or when it is exceptionally brilliant. One way to minimize this might be to identify one faculty as the facilitator and a different one as the “content” expert. In many ways, someone the students do not view as the main content person can make an excellent facilitator. The facilitator needs to know enough of the content to know how to direct the questioning (with pre-session guidance from the course director or “content” expert).

마무리 할 시간을 주기

Tip 6 Provide time for closure

잘 한 학생을 언급해주거나 부정확한 주장을 교정해준 학생을 언급해 줄 수도 있지만, 여전히 공식적으로 세션을 마무리짓는 것이 중요하며, 이 때 어려운 개념에 대해서 설명해주는 것이 좋다. 이러한 설명은 토론 중간이 아니라 토론이 다 마친 다음에 제공하는 것이 좋다.

One of the most important things that a facilitator should remember to do is provide time for closure at the end of the session. Providing closure after each question makes it difficult to manage time. However, highlighting a student’s excellent response or one who corrects an argument that was inaccurate can go a long way to bring satisfaction and clarity to the learning in the classroom and is not time intensive. Yet, it is still important to bring formal closure to conceptually difficult material at the end of a session as students often do not explicitly trust each other’s knowledge-base and prefer to hear from an expert faculty member or facilitator. Thus, adding closure to difficult concepts after but not during student discussions will assist in ensuring that students feel that the TBL process was valuable and that they learned the important “take-home” information.

학습자의 활발한 참여 유도하기

Enhancing active engagement of learners

The quality of a TBL session depends a lot on the facilitator’s ability to get the students engaged, but getting students to respond to questioning is difficult under most circumstances. There are several general suggestions to achieve classroom engagement. 

• First, be open and transparent about the intention and process of asking questions. 
• Second, create a safe environment where students can answer incorrectly without fear of ridicule or recrimination. 
• Third, consider using a randomization process (random team and random members within the team) to decide who to call upon. That way, students will not feel “picked” on. 

Even if the environment and processes are all in place, it can still be difficult to get students to participate. Tips 7–12 provide some strategies one can use to encourage reluctant learners to speak up.

학생들이 대답할 시간을 주기

Tip 7 Wait for students to respond to questions

It is easy for a facilitator to forget that students often become nervous and need time to gather their thoughts when asked to answer a question or to defend a response to a question during the RAT or application phases of TBL. Students do understand that during the classroom discussion phases of TBL they all must be prepared to defend the choices their teams have made, though it is reasonable to expect that they need some time to remind themselves what the issues were. Thus, when calling on students to respond, the facilitator is encouraged to give more time than he/she feels is necessary. Oftentimes, the silence following a question is not due to confusion over what was asked, but due to time they need to think about the way they wish to phrase their response. Other times, it is necessary for the student to consult his or her team mates as a reminder of why they answered something the way they did. Allowing up to 30 s may be necessary and expected. This is certainly difficult to do when time is limited and many questions are still remaining. If students feel that the environment is safe and tolerant, then they will engage more actively. Allowing for enough “dwell-time” after each student is called on should help in this regard.

중립적, 개방형 질문 사용하기

Tip 8 Ask neutral and open-ended questions

One of the best practices a facilitator should adopt is to ask an open-ended question to a specific person in the classroom that forces them to critique, analyze, justify, and explain their choice of answer (Silberman & Auerbach 2011). Questions that allow for a yes or no response will generally cease any further discussion. Questions that lead or direct a student towards a certain specific answer will also halt any further discussions, as the students perceive that they need to focus on the answer provided. Open-ended questions allow students to demonstrate their thinking. They also allow for easier follow-up questions, such as ones that force students to clarify or justify their responses. Questions which are more neutral, objective, and open-ended result in more informative and valuable responses from the entire classroom. A commonly used open-ended question is the “why” type of question, e.g., why did you chose this, why is this the better choice over other answers, or why not this option? Neutral open-ended questions will ensure active discussion occur and assist in the interrogation of students’ knowledge.

명확한 의미 전달을 위해서 바꾸어 다시 말하기

Tip 9 Rephrase or restate for clarity

As mentioned above, we want to create a safe environment where students can answer incorrectly without fear of ridicule or recrimination. One way of achieving that type of environment, which will help students willingly speak up, is to rephrase (with guidance towards the learning goal) when students present their team’s rationale to be sure the facilitator understands the answer. Sometimes this is necessary just because students may not speak in a clear and concise manner and often they are not heard by students in another corner of the room. Students are often not confident when addressing the classroom and are reticent to vocalize what they know or do not know. Students also frequently display “drift”, a process where they start reporting in a confident and audible fashion, but end in a barely audible and less confident tone. By summarizing and restating what was said by the student, a facilitator can keep learners engaged and ensure that everyone hears and that 

(1) 불명확한 정보를 명확하게 unclear information is clarified, 

(2) 복잡한 정보를 단순화해서 overly complex information presented is simplified, 

(3) 부정확한 정보를 통해 토론을 유도 incorrect information is stated (non-judgmentally) to elicit debate, and 

(4) 중요한 원칙을 반복할 수 있도록 principles can be repeated for best learning and retention. 

정답을 흘리거나 강의로 은근슬쩍 넘어가서는 안 됨

It is important to note that although the facilitator is repeating or rephrasing content for clarity often, during the facilitation process he or she is not ending discussion by “leaking” the correct answer or slipping into lecture mode and removing the students from the discussion. The process of succinctly repeating what a student has said benefits the entire classroom and goes a long way in making engaging and informative TBL sessions a success.

잘 하는 학생 찾기

Tip 10 Find the “student expert” in the room

그러나 '내용전문가'라는 칭호를 붙여주는 시점은 개개인의 생각을 충분히 탐색할 기회를 가진 후여야만 한다.

'내용전문가'학생을 찾기 어려울 수 있는데, 이 때는 "어떤 근거가 이 내용을 뒷받침할 수 있을까?" 또는 "혹시 이 문제를 해결하는데 도움을 줄 수 있는 사람?" 이라고 질문해볼 수 있다.
During any cross-examination of teams, it is important to recognize that somewhere in the classroom exists an expert who needs to be identified. Even the most difficult problems can usually be resolved by a student in the TBL classroom. And, only after attempts are made at finding that student expert should a facilitator either turn to a “content expert” for assistance or provide the answer to the problem him/herself. In fact, correcting or answering too early will result in shutting down any further conversation and interrupts self-enquiry from the students. Thus, as a rule, the facilitator should put on the “content expert” hat only after making sure that students get the opportunity to probe each other’s thinking as much as possible. It is often difficult to find the student expert in the classroom as students can be unsure of themselves and may not trust their colleagues either. One of the best ways to achieve this is to ask probing questions such as “What evidence supports this?” or “Can anyone assist us to resolve this issue?”

진행을 가로막는 불확실성이나 의견 불일치 중재하기

Tip 11 Ensure any lingering uncertainties or disagreements are addressed

학생들은 팀의 의견에 동의하지 않지만 무식해보이지 않기 위해서 동의하는 척을 한다고 함. 

잠재적 의견불일치를 찾아서 학생이 스스로 그 문제를 해결할 수 있도록 해야 한다.

Often there is significant divergence in the thinking behind a particular concept in the classroom and this may or may not be apparent due to the consensus building process inherent with TBL. Students will often report what they feel pressured by the team to report to avoid looking unknowledgeable even if they do not agree with their own team’s decision. Sometimes, a student will say that he/she felt one way but the team felt a different way about a particular question. It is important for facilitators to ask if there is someone in the classroom who agrees or disagrees with what was just reported and to do this often. It is those strong disagreements or uncertainties that produce the most learning as discussions tend to be more robust and passionate when people disagree. If students feel safe in expressing their opinions, then the level of classroom engagement can be allowed to reach its peak. Therefore, it is important to remember to seek out possible disagreements and try to get students to address them as they arise.

개개인 학습자에 대한 책임 강조하기

Tip 12 Hold each individual learner accountable

적절한 환경에서 잘 참여하지 않는 학생의 이름을 불러주는 것은 교수가 그 학생을 개인적으로 알고 있으며, 익명성 뒤에 숨을 수 없게 한다.

By creating an environment where each student knows they could be called upon at any time to respond or defend a team’s answer, you further ensure individual student and team accountability. As individuals, students will realize they must be prepared. As a team, they will try to make sure their team-mate represents the team well. It is advisable to avoid the assignment of a team spokesperson whenever possible and to remind students often that they are responsible for their team’s responses and choices. In addition, students must be reminded constantly that although their individual choices may vary, they should be prepared to defend their team’s decisions and explain their decision making processes to the classroom. Students who are constantly asking other team members for help may be called upon more often by facilitators who observe the classroom continuously. A facilitator who creates an effective “environment for engagement”, where the atmosphere is fair and tolerant and where people can feel free to make mistakes without fear of embarrassment, is best suited to create an environment of maximal participation. In large classrooms with more teams, it is even more important for facilitators to scan the room and identify individuals in the far corners who may be less engaged. Using students’ names when calling on them, in the right environment, will help student feel as though the facilitator knows them personally and they cannot hide by being anonymous.

It is also important to get participation from as many different individuals as possible and to avoid picking on the same vocal individuals repetitively. Again, there are no assurances that anyone can give to ensure that each student is giving 100% of their time, but active surveillance and constant vigilance can ensure that the participants remain as active as possible.

Conclusions

촉진자의 역할은 90:10의 법칙을 지키는 것이다. 90%의 시간동안 듣고, 10%시간동안 말하는 것이다.

As more interest is garnered around the use of TBL in the classroom, educators are interested in understanding how to manage the discussion phases to ensure complete and deep learning. A fair amount has been written concerning the various stages of TBL, the backward design elements, and the pedagogical elements of TBL (Michaelsen et al. 2008). Although some information has been published concerning facilitation in PBL (Leung et al. 2003; Yee et al. 2006), very little information is available on specific recommendations for effective facilitation of learning in the TBL classroom (Azer 2005). The 12 tips presented here are designed to assist faculty who facilitate discussions in the TBL classroom in two ways. 

• First, application of these tips will help to ensure maximal and consistent participation from students and provide continuity to sessions if adhered to by all faculties who teach different sessions. 
• Second, these tips are designed to assist the faculty who deliver TBL by recommending a platform that ensures a fair and safe learning environment, but one that holds students accountable to their own learning as well as that of fellow learners. A universal role of a facilitator is to observe the 90:10 rule – listen 90% of the time and talk 10% of the time (Silberman & Auerbach 2011). We consider the principle of listening more than you talk a sign of an effectively facilitated session.

Although the tips presented in this article are designed for faculty who teach in a TBL setting that promotes a maximal learning experience for their learners, many of these tips would work well in any classroom setting. For example, the principle of waiting to speak for a defined period of time after asking students a question is quite useful when using any teaching style. This same reasoning holds for asking neutral and open-ended questions. However, many of these tips are specific to TBL as this teaching style requires the management of multiple teams and its goal is to achieve maximal student engagement. For example, when providing a lecture to students, generally closure is automatically given. However, one of the important principles of TBL is the emphasis on self-discovery and self-directed learning. Thus, remembering to provide closure or a sufficient wrap-up in this environment is equally important. The same can be said of the separation of the content delivery from the facilitation of learning. The lecture-dependent teaching style requires the content expert to be in the content delivery mode most of the time. However, in PBL and TBL, the students must make their own discoveries first and they must identify the important content and understand it before a content expert clarifies and adds to the content.

Much of facilitation in the TBL classroom is a strategy and can be learned. These tips can play a role in this process. However, it is clear that there is some “art” to the practice of being a good facilitator. This art is hard to teach. Experience and practice is the best way to learn the “art of facilitation”. Some faculty may find facilitation in the TBL classroom a challenge. Others will find it intuitive, fun, and exciting and may do well without much training. However, in either case, we hope that faculty either new to facilitating or those who have been doing this for some time find these tips for effective facilitation in the TBL classroom most useful and practical.

Med Teach. 2015 Feb 10:1-6. [Epub ahead of print]

Twelve tips for facilitating team-based learning.

팀바탕학습에서 동료평가를 어떻게 시행할 것인가?

How to Administer the Peer Evaluation in Team-Based Learning

허선

Sun Huh

한림대학교 의과대학 기생충학교실과 의학교육연구소

Department of Parasitology and Institute of Medical Education, College of Medicine, Hallym University, Chuncheon, Korea

팀바탕학습 도입이 우리나라 의과대학에서 점점 늘어나고 있다. 그런데, 이런 학습 방법으로 진행하였을 때 성취도 평가를 어떻게 할 것인지는 의대 교원에게 고민거리이다. 최종 학습자 개인 성취도 평가는 지필고사, 컴퓨터고사, 임상수행평가, 구두시험 등으로 하게 되지만 팀바탕학습에 특화된 평가 도구를 찾기 쉽지 않다. 팀바탕학습을 수행하고 해당 내용 성취도 평가는 대개 위와 같은 평가 도구, 팀바탕학습 중 individual readiness assurance test, group readiness assurance test, further application에 성취도와 참여도에 따른 교원의 평가, 그리고 동료평가 세 가지를 활용한다. 이 세 가지에 점수 배분을 어떻게 할 것인지는 담당 교원이 선택하게 된다. 세 가지 가운데 전통적인 평가 도구와 팀바탕학습 과정 중에 평가는 객관적인 자료가 나오므로 교원이 수행하고 평가하는 데 큰 어려움이 없으나, 동료평가는 참 쉽지 않다.

우리나라 의대생에 국한된 것만이 아니지만 우리나라 사람은 평가하는 데 주관이나 직관이 포함되는 것에 매우 저항이 크다. 그렇기에 정부에서 시행하는 행정, 외교 분야 공무원 선발이 철저히 시험 성적에 달려 있다. 의대생도 동료평가를 매우 부담스럽게 여기기는 마찬가지이다. 한림의대에서 의학과 1학년 대상 팀바탕학습에서 동료평가를 준비도, 공헌도, 타인 존중도, 유연성 등 4개 항목으로 구분하여 시행하여 보았더니 동료에게 동일한 점수를 주기 위하여 4개 평가 항목을 골고루 나누어 점수를 주어 결국은 같은 점수를 동료가 받도록 하는 행태가 나타남을 알 수 있었다. 즉, 팀바탕학습에서 구성원의 준비도 등은 모두 다르고 다들 알지만 그것을 실제로 표현하는 것에 익숙하지 않다. 우리 사회가 같은 집단에서 구성원 사이 동료평가를 한 경험이 많지 않고, 보통교육에서부터 그런 훈련이나 과정이 거의 없었고 단지 경쟁은 시험을 통하여 하는 데 익숙한 문화 때문이라 여긴다. 더구나 이 동료평가를 개인 학습 성취도 평가에 반영한다고 하면 더욱 긴장하기 마련이다. 그렇지만 동료평가는 앞으로 의사로서 살아가는 동안 내내 부딪치는 것이고 조직 발전과 개인 발전에도 필수 과정이므로 이런 평가 과정에 익숙하여야 하며, 정확하게 수행할 필요가 있다. 또한 동료평가가 결국 집단과 개인에게 도움을 주는 것임을 깨달으면 어려움 없이 수행할 것이다.

과연 어떤 방법을 사용할 것인가? 교재에 보면 The Michaelsen Method, Fink Method, Combination of Michaelsen and Fink Method, Koles Method, Texas Tech Method 등이 있다. 이 교재는 이미 국문으로 번역하여 쉽게 읽을 수 있으므로 서점이나 도서관에서 쉽게 찾을 수 있을 것이다[1]. 각 학교에서 적절한 것을 선택할 수 있다. 여기서는 한림의대 의학과 1학년 수업에 사용한 예를 소개한다(Appendix 1). 이 표에서는 모든 조원에 대하여 상대평가를 하여 순위를 매기도록 하였다. 이 방법은 철저한 상대평가를 한 것인데 앞으로 효과가 있는지 점검이 필요하다. 이런 순위 평가 이외 점수를 주도록 하고 대신 점수에 차별을 주도록 하고 총점을 같도록 하여 상대평가를 하는 방법도 있다. 대학마다 각각 다른 동료평가 도구를 사용할 것이고 결과를 보고 개선하여 나가면 충분할 것이다. 동료 평가를 얼마나 자주 할 것인가도 매번 할 수도 있고 전체 수업기간 중 1~2회 할 수도 있으므로 역시 교원이 판단하여 시행하면 충분하다. 팀바탕 학습에서 동료평가는 아직 개선할 내용이 많고 특히 직업전문성(professionalism)을 기를 때 중요한 평가항목이므로 더 연구가 필요하다.

의학교육에서의 팀 바탕학습 운영과 효과성에 대한 고찰

Review on the administration and effectiveness of team-based learning in medical education

허예라1, 조아라2, 김 선

Yera Hur1, A-Ra Cho2 and Sun Kim2

1건양대학교 의과대학 교수개발 및 멘토링센터, 2가톨릭대학교 의과대학 의학교육학과

1Faculty Development & Mentoring Center, Konyang University College of Medicine, Daejeon, and 2Department of Medical Education, The Catholic University of Korea College of Medicine, Seoul, Korea

지식의 전달이 중심이었던 의학교육은 1985년 미국의과대학협회(American Association of Medical Colleges)의General Professional Education of the Physician 보고서가 발표되면서 성과바탕(outcome-based) 의학교육으로 전환하고, 학생들이 졸업 후 최소한 갖추어야 할 성과를 규정하고 새로운 교육목표를 설정하기 시작하였다[1]. 새로운 의학교육의 목표는 세 가지로 함축해 볼 수 있는데, 쏟아지는 의학지식의 홍수 속에서 진단과 치료에 필요한 정보를 수집하고활용할 수 있는 자기주도학습능력, 협력진료 환경 속 효과적인 팀워크 능력, 기초와 임상의학을 통합한 비평적 사고 능력의 함양이 그것이다[2]. 새로운 교육목표는 자연스레 교육방법의 변화를 요구하였다. 이러한 패러다임의 변화를 수용하여 그 동안 대부분의 교육과정을 전통적인 강의 중심으로 운영해 오던 의과대학들은 팀 바탕학습(team-based learning,TBL) 도입을 위한 논의를 활발히 진행해 오고 있으며, 그에따라 TBL을 도입하고 있는 의과대학의 수는 점차 증가하고있다[3]. 특히 외국의 대표적인 의학저널 검색엔진 PubMed에 “team-based learning”, “TBL” 키워드를 입력하면 2000년1월부터 2013년 10월까지 70편 이상 검색되는데 이 가운데50편 이상이 2010년 이후 발표되었으며, MedEdPORTAL에는 2013년 10월까지 80여개의 TBL 모듈과 자원이 공유되고있다[4]. 이러한 수치는 최근 TBL에 많은 관심이 집중되고있으며, 이미 다양한 의학교육 분야에서 많은 경험이 축적되고 있음을 시사해 준다. 실제로 이미 60여 개 이상의 의학교육기관들이 의과대학 교육과정(undergraduate medical education),전공의 교육수련과정(graduate medical education),의사의 평생교육과정(continuing medical education)에 TBL을 도입하고 있으며, 성공적인 교육 경험을 바탕으로TBL 운영과 효과성에 대한 연구 결과를 지속적으로 발표해오고 있다[5,6]. 반면 국내에서는 TBL 도입 움직임이 여전히미비한 수준이며 연구 현황도 열악하다. 이는 국내에서 TBL을 도입하고 있는 의과대학의 수가 매우 제한적이라는 현실에 기인한다. 이러한 배경에서 이 연구는 외국의 선행 연구를바탕으로 TBL의 기본 원리와 진행 단계를 살피고, 의학교육에서의 TBL 운영 유형과 효과성에 대한 분석을 통해 국내의의학교육에서 TBL의 도입을 위한 과제가 무엇인지 도출해보았다.

TBL 기본 원리와 진행 단계

TBL은 ‘지식의 전달(knowledge transmission)’이 아닌 ‘지식의 활용(knowledge application)’에 목적을 두고 있는 교수법이다[6]. 이 목적을 달성하기 위해 TBL은 두 가지 변화를 시도한다. 

• 첫 번째 변화는 교수와 학생의 역할 변화이다. TBL에서 교수자는 ‘지식의 전달자’가 아닌 교육과정 개발자,수업 운영자이자 ‘학습의 촉진자’로, 학습자는 지식의 암기와이해를 넘어 스스로 정보를 탐색, 수집하고 이를 분석하여 종합적인 사고를 바탕으로 문제를 해결하는 ‘학습의 주체자’로역할을 하게 된다[7]. 
• 두 번째 변화는 수업 구조의 변화이다(Fig. 1)[8]. 일반 강의와 달리 TBL의 수업 구조는 팀 활동을바탕으로 실제 의료현장에서 부딪힐 수 있는 문제를 해결하는 과정에 수업 시간의 대부분을 할애한다는 핵심적인 차이가 있다.

이러한 TBL의 원리는 자기주도학습(self-directed learning),준비도 확인(readiness assurance), 적용 학습(applicationactivities), 평가(assessment)의 단계를 거치며 실현된다. 수업 준비에서부터 평가까지의 세부 진행 과정과 교수의 역할을 살펴보면 다음과 같으며(Fig. 2) [9], TBL 수업을준비할 때 Michaelsen &Seet [10]이 규명한 성공적인 TBL을 위한 4가지 핵심 원칙을 고려한다면 보다 효과적인 운영이가능하다(Fig. 3).

TBL 운영 유형

선행연구를 검토한 결과 TBL은 크게 두 가지 유형으로 운영되고 있었다. 연구에 따라 사용하는 용어에는 차이가 있지만, 이 연구에서는 의미를 명확하게 전달하기 위하여 기본적인 TBL 단계에 따라 운영하는 classic TBL과 다른 교수법과통합 운영하는 adapted TBL로 구분하여 기술하고자 한다.

1. Classic TBL

Haidet et al. [6]은 TBL 기본 구조를 

1) pre-class preparation,

2) assurance of readiness to apply learned concepts,

3) application of content through group problemsolving activities 

로 제시하고 있다. Classic TBL 유형은TBL 1세션에 3단계 기본 구조를 모두 포함해 운영하는 형태로, TBL을 도입하고 있는 대부분의 대학이 이 유형을 채택하고 있으며 일부는 교육과정에 적합하게 수정하여 적용하고있었다. Classic TBL로 교육과정을 운영하기 위해서는 계획단계에서부터 TBL 도입 여부를 사전에 결정해 새롭게 설계하여야 한다. 한 예로 Fujikura et al. [11]은 TBL 교수개발워크숍을 수료한 20명의 교수를 투입하고 특정 교수의 개별수업 차원이 아닌 전체 교육과정 차원에서 임상의학 TBL 과정을 설계하였으며, 총 15개의 TBL 세션(3 hours/1 session)으로 운영하여 교육내용의 연속성을 확보하였다. 또한Masters [12]도 총 11개의 TBL 세션(2 hours/1 session)으로 의료정보학∥과정을 운영하여 학생들이 충분히 TBL을 경험할 수 있도록 하였다. 이처럼 TBL 도입을 계획했을 때 가장 중요한 것은 충분한 교육 시간의 확보이다. 이러한 맥락에서 Wright State University 의과대학은 1, 2학년 기초의학21개 과정과 3, 4학년 임상실습 8개 과정 가운데 각각 18개,5개 과정 수업의 대부분을 TBL로 운영하고 있어 시사하는 바가 매우 크다. 이미 선행연구에서도 TBL에서 기대되는 교육적 효과는 학생들이 지속적으로 TBL을 경험하였을 때 실현가능함을 강조하고 있다[13].

2. Adapted TBL

Adapted TBL 유형의 대표적인 예는 PBL과의 통합 운영형태이다. 실제로 Fujikura et al. [11]은 TBL이 PBL을 진행할 때 사전 지식이 부족하여 토론이 진행되지 않는 한계를 극복할 수 있는 효과적인 촉진 전략이라고 평가하였으며,Anwar et al. [14]은 TBL이 PBL 과정 중 학생들의 학업성취를 향상시키는 데 유용한 전략이라고 보고하였다. 특히Abdelkhalek et al. [15]은 PBL과 TBL을 결합한 구체적인수업 모형을 제시(Fig. 4), TBL이 적은 자원으로 많은 수의학생을 충분히 준비시키는 데 효과적이기 때문에 발견 학습(discovery learning)을 지원하는 전략으로 활용되기에 적합하다고 강조하였다.

결론

교육의 주체는 학생이다[16]. TBL은 그 동안 교수 중심으로 이루어져 왔던 의학교육에서 학생 중심 교육을 실현해 주는 구조적인 전략이란 평가를 받고 있는 교수법이다[17]. 이연구는 선행연구를 통해 TBL의 기본 원리와 진행 단계를 정리하고, TBL 운영 유형을 살펴보았다.

그 결과 TBL은 이미 다양한 의학교육과정에 도입되어 긍정적인 성과를 거두고 있음을 발견할 수 있었으며, 특히 TBL을 통해 기대되는 교육적 효과는 학생들이 TBL을 지속적으로 경험했을 때 가능함을 확인할 수 있었다. 이는 국내의 의학교육에 TBL이 효과적으로 정착하려면 가장 먼저 TBL 운영에 필요한 충분한 시간을 확보한 후 기존 교육과정을 분석하여 재설계하는 작업이 우선되어야 함을 시사해 준다. 만약 대학의 여건상 교육과정의 재설계가 불가능하거나 많은 시간을TBL에 할애하는 것에 어려움이 있다면 PBL과 병합하여 별도의 시간 확보 없이 기존의 교육과정 틀 안에서 TBL을 운영하는 하는 방법을 제안해 본다. 이 외에도 Brich [18]와 같이정규 교육과정의 보조 프로그램(supplemental program)으로 TBL을 운영하여 TBL의 교육적 효과를 기대해 보는 것도대안이 될 수 있다. 또한 효과적인 TBL 운영을 위해서는computer based test system, 원형 테이블 등 팀 학습에 적합한 교육환경을 구축하고[19,20], audience response system과 같이 상호작용을 촉진할 수 있는 도구를 활용하려는 노력도 필요하다[11].

지금까지 의학교육에서 TBL의 교육적 효과에 대한 결과를 제시하고 있는 선행연구에 따르면 TBL은 학생들의 학업성취도를 향상시키며, 교수와 학생들의 수업 만족도를 높여 준다.구체적으로 Burgess et al. [21]은 TBL이 대규모 학생을 대상으로 팀 학습을 가능하게 하며 지식의 습득에 효과적인 방법이라고 평가하였고, Tan et al. [22]은 passive learning 교수법을 사용한 수업과 TBL로 진행한 수업의 학생 성적을 비교하여, Vasan et al. [23]은 TBL을 도입하기 전과 후 학생들의National Board of Medical Examiners 점수를 비교하여TBL이 다른 교수법에 비해 학생들의 학업성취도 향상에 도움을 주고 있음을 증명하였다. 또한 TBL은 학생 간의 성적편차를 줄이고, 과목 낙제율을 낮추며[13] 특히 성적이 낮은그룹 학생들의 학습을 돕는다[24]. 이처럼 TBL 연구는 대부분 학생과 교수의 만족도, 시험 성적을 결과 변수로 설정하고교육적 효과를 측정하지만 이 외에도 구조화된 도구를 사용하여 학생들의 정서지능(emotional intelligence)과 임상적추론능력(clinical reasoning ability)을 측정하여 의미 있는결과를 보여준 연구도 있다. 특히 Okubo et al. [25]은Problem-Solving Ability Test (P-SAT)을 실시하여 TBL을 통해 학생들의 임상적 추론 능력이 향상되었음을 확인하였고, Borges et al. [26]은 Workgroup Emotional Intelligence Profile-Short Version (WEIP-S) 검사 결과를 제시하며 TBL이 학생들의 정서 지능 발달에도 긍정적인 영향을주고 있음을 확인시켜 준다. 또한 Bick et al. [27]은 TBL이기초의학과 임상의학의 통합을 가능하게 하고 학생들의 의사소통기술과 리더십 능력의 향상을 가져온다고 보고하였으며,Parmelee &Hudes [28]는 의학전문직업성(medical professionalism),피드백 역량을 향상시키고 평생학습자로 성장하기 위한 훈련을 제공해 준다고 하였다.

앞서 논의한 바와 같이 TBL은 다양한 교육적 효과를 가져오며 교수와 학생 모두 높은 만족도를 나타내고 있는 교수법이다[18]. 하지만 선행연구를 통해 확인된 결과 외에 TBL을지속적으로 경험했을 때 기대되는 다양한 교육적 효과에 대한 근거는 아직 제한적이다. 따라서 앞으로 TBL이 국내 의학교육 장면에 적극적으로 도입되어 폭넓게 활성화되기 위해서는 국내 의학교육에서의 효과성을 확인하고 구체적인 적용방안을 제안하는 연구들이 지속적으로 발표되어야 할 것이다.

또한 TBL에 대한 부정적인 인식에 대해서도 충분한 논의가 이루어져야 한다. 특히 기본 개념만 확인하고 지나가야 할readiness assurance 과정에 불필요하게 많은 시간이 소모되어 실제 적용 학습에서 깊이 있는 토론을 진행할 시간이 부족했으며 동료를 평가하는 것이 어렵고 불편했다는 의견, 팀 학습을 중심으로 진행되는 수업이 자신의 학습 스타일에 맞지않아 부담스러웠다는 학생들의 의견에 주목할 필요가 있다[29,30]. 따라서 보다 효과적으로 TBL을 운영하기 위해서는각 단계의 목적에 부합되도록 적절하게 시간을 배분해야 하며, 동료평가의 공정성을 확보하고[29] 학생들의 학습 유형[30]과 개인적 특성[31]을 고려해 팀을 구성하는 등의 노력을기울여야 할 것이다. 그 가운데 동료평가는 TBL의 핵심이 되는 과정이며 더 나아가 의사에게 필수적인 역량임에도 불구하고 이에 대한 저항이 커 교수에게는 부담이 되는 평가 방법일 수 밖에 없으므로[32], 객관적인 평가가 가능한 구조화된양식을 개발하고 학생들을 대상으로 실제적인 평가자 훈련을제공하는 등 공식적인 평가 도구로 자리매김할 수 있도록 하기 위한 고민이 시급하다.

결론적으로 TBL은 현재 의학교육이 목표로 하는 학생 중심의 통합교육을 실현시켜 주는 교수법이다. 추후 보다 많은국내의 의과대학이 TBL을 도입하여 다양한 교육적 효과를거두고, 그 경험을 바탕으로 적극적인 연구가 진행되어 TBL이 의학교육의 질 향상을 도모할 수 있는 경쟁력 있는 교수법으로 자리매김할 수 있기를 기대한다

효과적인 TBL을 위한 12가지 팁(Medical Teacher, 2010)

Twelve tips for doing effective Team-Based Learning (TBL)

DEAN X. PARMELEE1 & LARRY K. MICHAELSEN2

1Wright State University, USA, 2University of Central Missouri, USA

의학교육학자들이 인지한 두 가지 중요한 현실. 여러 질병에 대해서 줄줄 읊을 수 있는 것과 실제 환자를 보고 빠르게 진단내리는 능력은 별개의 것이다. 의과대학생은 이 두 가지를 모두 알아야 한다.

Medical educators have long recognized two important realities. 

• One is that being able to recite all the subtle differences between one form of a disease and another is a very different kind of knowledge than being able to quickly diagnose the correct form of that disease suffered by a real, living patient. 
• The other is that medical students must master both kinds of knowledge.

전통적으로 학생들은 이 두 가지 서로 다른 지식을 서로 다른 시간에 서로 다른 상황에서 다뤄왔다. 

In traditional medical education, students were exposed to the two different kinds of knowledge at different times and in different settings. The content was typically taught in lecture-based courses and, later (some years later) students learned to use the content during their time in clinical rotations.

그러나 배운 내용을 학습할 수 있는 환경을 뒤로 미뤄두는 것은 '성인이 가장 잘 배우는 방법'과 맞지 않는다. 이에 의학교육자들은 학생이 배우는 내용과 그것을 적용할 수 있는 환경을 최대한 근접하게 만드는 방법을 개발해왔고, PBL이나 사례발표가 그 예다.

Delaying students’ opportunity to learn to use the content, however, does not fit well with what we now know about how adults learn best – the kind of learning that both ‘sticks’ and can be transferred to novel situations. As a result, medical educators have experimented with a number of approaches for enabling students to more closely connect the content and concept acquisition with its application – e.g. problem-based learning (PBL), case presentation.

(...)

TBL진행 개요

For a course with TBL as part of its learning activities, 

• students are strategically organized into permanent groups (for the entire term of the course) and the course content is organized into major units (typically five to seven). 
• Before each in-class event, students must study assigned materials because each module begins with the readiness assurance process (RAP). The RAP consists of a short test (over the key content and concepts from the readings or other activities, e.g. dissection) which students first complete as individuals, then they take the exact same test again as a team, coming to consensus on each question. 
• Students receive immediate feedback on the team test and they then have the opportunity to write evidence-based appeals if they feel they can make valid arguments for their answers to questions which they got wrong. 
• The final step in the RAP could be a ‘lecture’ (usually very short and always very specific) to enable the instructor to clarify any misperceptions that become apparent during the team test and the appeals, but also could be a between-team discussion about why the selected correct answers are best – fielded by the instructor. 
• Once the RAP is completed, the remainder (and the majority) of the learning module is spent on in-class activities and assignments that require students to practice using the course content by solving challenging problems.

12개의 팁

TWELVE TIPS

좋은 교과목 설계로부터 출발하라

Tip 1: Start with good course design

교과목의 주요 이슈는 다음과 같다.

TBL is an instructional strategy that works best when it is integrated tightly with a course's design. It can be the primary mode of instruction or work alongside other learning activities, i.e. focused lecture, service learning, self-directed online tutorials. We recommend using Dee Fink's Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses (2003) for guidance in defining a course's (or curriculum's) contextual issues, 

• goals, 
• assessments, 
• learning activities, and 
• feedback mechanisms. 

기존 강의의 한두시간을 대체해서 시도해볼 수 있다.

Often, instructors will ‘try out’ a TBL module or two in an existing course, either replacing a set of lectures or small group sessions that had required recruiting and herding many faculties. This is a valid way to gain experience with how to implement it, but, usually, it is hard to incorporate the peer evaluation component since the number of meetings will be few.

TBL과정과 모듈을 만들 때 후향 설계(backwards design)를 해보라

Tip 2: Use a ‘backwards design’ when developing TBL courses and modules

후향적 설계란 "이 시간이 끝난 후 학생들이 무엇을 할 수 있기를 바라는가?"를 스스로 질문하는 것이다.

With backwards design (Wiggins & McTighe 1998) the first question to ask yourself is, ‘What do I want my students to be able to DO by the end of this unit of study?’ Whether designing a single TBL module for a unit of study, e.g. Starling's Law and cardiovascular physiology, or a series of modules that form the basis of an entire course, clarify what you want the students to be able to do by the end of the module or course. For example, a goal for a module in physiology/pharmacology focused on Starling's Law would be for the students to be 

• able to apply their understanding of Starling's law to accurately interpret physiologic data from a case of congestive heart failure, 
• explain how Starling's Law governs which findings, 
• predict which pharmacologic agent will affect specific components of heart function. 

(...)

내용에 대해 잘 아는 교수자들이 스스로 이 질문에 답을 하기는 어려울 수 있다.

This single question is often the hardest one for instructors who are ‘content-driven’ to ask themselves. There is just so much ‘content’ that we feel our students must know before they can make use of it – but, TBL provides a way to have them master the content while they are applying it and get feedback on how well they are ‘getting it’ as they go.

학생들이 도달할 수 있으면서 교수자(그리고 학생이) 어느 정도 도달했는가를 알 수 있는 모듈 구성

Tip 3: Make sure you organize the module activities so that students can reach your learning goals and you (and they) will know that they have done it

어떤 활동을 할 것인가를 구성해야 하는데, 일련의 문제를 제시해서 학생들의 사고과정을 이끌어나가는 수단으로 활용하고 싶은 욕망을 억제해야 한다. 학생들이 스스로 어려운 선택을 하고, 함께 작업하여 그 개념을 익혀하도록 해야 한다.

After clarifying what you want your students to be able to do by the end of the module, the next step in backwards design is creating a group application exercise. This should be a problem that requires students to use all of the preparatory knowledge and their team's brainpower to analyze, interpret, and then commit to a choice or a decision. Further, you should avoid the temptation to ask a series of questions as a means of ‘leading students through the thinking process.’ It is far better to require them to make a difficult choice and let them work together to master the concepts and to discover and internalize the relationships between them in the process of coming to a conclusion.

무엇을 할 수 있게 할 것이며, 그것을 어떻게 평가할지를 결정했다면 그 문제를 해결하기 위해서 수업 전에 무엇을 해와야 할지를 결정해야 한다. 또한 RAT 문제를 출제해야 한다. (이를 퀴즈라 부르지 말라)

Once you have decided what you want students to be able to do and how you will assess whether or not they can do it, the next two steps in backwards design are identifying what content elements the class must master before they are ‘ready’ to solve the problem (i.e. the information that the students need to learn outside of class to be prepared for the module) and write the questions for the readiness assurance test (RAT) (and do not call it a quiz – its purpose is readiness assurance and you should emphasize its role by the terms you use in talking about it).

심도있는 사고, 참여, 학습내용 중심 토론을 위한 학습활동을 만들라

Tip 4: Have application exercises that promote both deep thinking and engaged, content-focused discussion

TBL성공에 가장 중요한 것은 학생들에게 만들도록 한 과제가 무엇인지이다. 내용이 무엇이든 '긴 결과물을 내라'라고 하면 학생들은 그것을 서로 분담하고, 학습도 잘 안 이루어지게 되고, 부정적인 인식을 하게 된다. 잘 설계된 과제를 내면 학생들은 서로에게 배우면서 상당한 자신감을 얻게 될 것이다.

Over the years, we have come to realize that the single most important aspect of successfully implementing TBL is what your assignments require students to create. Whatever the content, if you ask them to produce a lengthy document, they will divide up the work which, in turn, will reduce learning and, all too often, will result in negative feelings about their peers and skepticism about working in a group. On the other hand, we have learned that, by using well-designed assignments, students will both learn from each other and develop a great deal of confidence in the value of working in a team.

좋은 과제는 각 단계(혼자 하기 - 팀 내에서 하기 - 팀 사이에서 하기)에서 다음의 네 가지 S를 만족하는 것이다. 

The key to designing effective assignments is ensuring that what students are asked to do is characterized by 4 S's at each of the stages in which they engage with the course content – working alone, working within their team, and working across teams (i.e. whole-class discussion). The 4 S's are:

• 실제로 중요한 문제 Significant Problem
• For a successful group application exercise, select or create a problem that the student can readily recognize as the kind of problem that will be encountered in ‘real life,’ make it Significant. In medical education, this is easy – there are an infinite number of patient cases that are rich with data to be interpreted, decisions to be made. But, there has to be a clear link between the content that underlies the exercise and its application. At the conclusion of the hypothetical module on Starling's law/physiology, you want to hear your students talking about how the basic principles of the law are applicable to understanding cardiac contractility in stress situations and how to approach interventions. In addition, the answers to these questions should never be discoverable in a text or article or lecture notes – they can only come from team members collaborating to figure them out.

• 모두에게 동일한 문제(서로 모두 같아야 하며, 따라서 학생은 문제 선택권이 없다.) Same Problem
• With TBL, all of the small groups must be working on the Same Problem. If you assign different problems to different small groups, students are not accountable to each other because you lose the benefit of having any semblance of a robust discussion (and learning!) between-group discussion of the problem. Further, if you allow groups to choose their own problem, they are not even accountable to you – unless you are willing to do the research that you hope they would do.

• 구체적 문제(이 환자에게 가장 좋은 약은? < 이 환자에게 가장 좋은 약을 썼을 때 영향을 받는 신경전달물질의 구성은?) Specific Choice
• When your assignments require students to agree on a specific choice, the only way they can accomplish the task is by working together to critically appraise a situation, examine the existing evidence, and make a professional judgment. Further, the more specific the question, the better the learning. For instance, if your module was about depression and pharmacologic interventions, a good question would be ‘Identify the set of neurotransmitters that are affected by the best drug choice for this patient’ and not ‘What would be the best drug for this patient’ because a more specific question requires a deeper analysis.

• 동시에 보고하기 Simultaneous Report
• You create an important ‘moment of truth’ when all the small groups are asked to post their responses to a question at the same time. Two things happen as soon as students realize that the choice they will be making will be open to challenges from other groups. 자신들의 의견이 다른 사람들의 공격을 받을 수 있을 것이라는 것을 깨달은 순간 다음의 일이 발생한다.
• 아군 vs 적군의 구도가 되어 그룹의 응집력이 올라간다.One is that, because of the potential of an ‘us versus them’ situation, group cohesiveness increases. 
• '정답을 내놓지 못해도' 숨을 곳이 없으므로 그룹 내 토론에 좀 더 활발히 참여하게 된다. The other is that students are far more engaged in the within-groups discussion because they realize that they would not be able to hide if they do not ‘get it right.’ 

• 각 그룹이 답을 도출한 과정을 바탕으로 하여 토론을 촉진할 수 있다.
  In addition, by engaging students exploring how they arrived at their respective answers, you can readily create a class discussion that is far more informative to you and your students than asking, ‘Somebody say what they think about thus-and-such.’

두 가지 안 좋은 점이 있다.

We have also learned two lessons – sometimes by sad experience – about the 4 S's. 

• 4S중 하나가 충족되지 않으면 토론의 강도와 그에 따른 학습이 저해된다. One is that failing to do any one of the 4 S's substantially reduces both the intensity of class discussions and the resultant learning. 
• 4S중 두 개가 충족되지 않으면, 학습은 거의 일어나지 않으며 점수를 깎이지 않으려는 것이 학생이 과제를 수행하는 유일한 이유이기 때문이다. The other is that, if you fail to do any two of the 4 S's, learning is minimal and pretty much the only reason that students are willing to complete the assignment is that it will have a negative impact on their grade.

RAP의 중요성을 과소평가하지 말라

Tip 5: Do not underestimate the importance of the RAP

RAP의 역할: 피드백, 피어티칭, 학생들의 gap 파악, 필요할 경우 강의

The RAP is designed to link students’ advance preparation to the group application exercises and provides a remarkable and powerful opportunity for individual feedback and peer teaching within the teams. In addition, the RAP lets you (and the students) know if you need to address gaps in their understanding. If the content area is particularly difficult, e.g. autonomics, odds ratios and predictive values in critical appraisal, liver pathology, then the RAP should be separated in time from the group application exercise so that the instructor can give corrective feedback and/or provide additional input before they begin to tackle the group application exercise. However, you do not have to cover everything – only what you (and the students) know they need help with.

RAP가 잘 이뤄지면 다음과 같은 장점이 있다.

The RAP, when done well, unfailingly produces five priceless outcomes even though it typically uses only a fraction of the overall class time (usually about 25–30%) for any given unit of instruction. These are:

• Effective and efficient content coverage.
• Development of real teams and students’ interpersonal and teamwork skills.
• Students gain an experience-based insight about the value of diverse input.
• Development of students’ self-study and life-long learning skills.
• Class time during which you can provide the content expertise to ensure that students develop critical thinking skills.

99.9% 이상 팀 내에서 가장 점수가 높은 학생의 점수보다 팀 점수가 더 높으며, 가장 낮은 점수를 받은 팀의 점수도 전체 반에서 가장 높은 점수를 받은 학생의 점수보다 높다.

In addition, data from the RAP provides data that definitively answers the question of whether or not individuals are likely to be held back by working in teams. Based on data from the past 23 years of using TBL (Michaelsen and Parmelee, unpublished), teams will score higher than their own very best member 99.9+% of the time1 and the most common outcome is that the worst team score will be higher than the highest individual score in an entire class.

종종 발생하는 실수에는 다음과 같은 것들이 있다.

Unfortunately, we have seen some instructors miss out on part or all of these valuable outcomes because they have, for whatever reason, decided to: 

• (1) skip either the individual or the group component – or both; 
• (2) use questions that are merely designed see if the students did the reading, e.g. asking ‘picky’ or meaningless questions unrelated to the objectives of the module; and 
• (3) view this process as a way of getting another assessment for their course grade. The RAP is not just another ‘quiz’ and neither instructors nor students will reap its many potential benefits if it is treated as such.

왜 TBL을 사용하는지, 이전 그룹 학습과 어떻게 다른지에 대해 학생들에게 설명하기

Tip 6: Orient the class to why you are using TBL and how it is different from previous experience they may have had with learning groups

대부분은 경험도 없고, 있더라도 형편없이 설계된 과제를 해야 했을 것이다. 혹은 잘 참여하지 않는 그룹원들을 데리고 뭐든 해보려고 고군분투 했었을 것이다.

Most students will not have had a classroom experience like TBL. In fact, the majority of their experience with group work will have been struggling to complete poorly designed assignments that forced them into the uncomfortable position of having to choose between doing more than their fair share or risk getting a bad grade and/or having to deal with difficult group members just to get anything done at all.

이러한 걱정은 실제하는 것이며, 왜 TBL을 사용하며 어떻게 TBL이 설계되었는지를 제대로 이해시키지 못하면 학생들의 적극적 참여를 이끌어내기 어려울 것이다. 최소한 교과목의 목표를 제시해야 하며, TBL에서는 전통적인 방법과 어떻게 다르게 그 목표를 달성하게 될 것인지를 설명해야 한다.

These concerns are real and must be addressed or you will have a difficult time getting student buy-in unless students understand both why you are using TBL and how TBL is designed to avoid the problems that they, all too often, have come to expect are a normal outcome from doing group work. At a minimum, you need to outline your course objectives and provide an explanation of how they would be achieved in a traditionally taught course versus how you will achieve them by using TBL. 

학생들의 TBL에 대한 이해를 도와주기 위해서는..

Other suggestions to help them understand and accept TBL include: 

(1) giving a practice RAT (many use the course syllabus as the ‘subject matter’ for the test); 

(2) engaging them in the process of determining the grading system for the course (Michaelsen et al. 2004) and, throughout the course; and 

(3) reminding them about the benefits they are experiencing along the way.

TBL의 핵심으로서 '책임'을 강조하기

Tip 7: Highlight accountability as the cornerstone of TBL

TBL의 성공에는 학생, 팀, 교수의 책임이 중요하다.

The cornerstone of success of TBL is that the natural outcome of its processes is that individuals, teams, and the instructor are immediately and clearly accountable for behaving in ways that promote learning. 

• Students are accountable for coming to class, preparing before they come, and investing time and effort working in their team. 
• The instructor is accountable for providing students with the cognitive foundation they will need to be ready to tackle the kinds of problems they will face in medical practice and giving them opportunities to practice developing their application skills.

When TBL is fully employed, the vast majority of students are prepared, come to class, and engage each other in productive ways as they work together. As a result, even the students who start out with a skeptical attitude because of past negative experiences with learning groups will eventually embrace TBL – ‘Finally, hard work as an individual and hard work as a group pays off.’

교수의 책임: 일부 학생들은 교수가 다른 수업에서 하듯 '가르치지' 않는다는 인상을 받기 시작할 것이다. 만약 교수가 여기서 책임을 다하지 않으면(4S의 활용, 학생에게 TBL의 장점에 대해 충분히 설명하기) 그러한 의심이 더 커지고 분노를 느낄 것이다.

As for the accountability of the instructor, some students will inevitably start out with the impression that he/she is not ‘teaching’ as in other classes, i.e. using lectures to state what will be on the final exam, and, worse, we (the students) are having to do all the work. Further, if the instructor is not following through with his/her side of the bargain – doing a good job of: (1) providing students with the opportunity to practice using well-designed applications assignments (i.e. using the 4 S's) and (2) reminding students of the benefits that they are getting – then the doubts and the resentment are likely to persist.

오답에 대한 의견을 말할 수 있는 기회 주기

Tip 8: Providing a fair appeals process will inspire further learning

문제가 혼란스럽게 기술되었거나(따라서 학생은 문제가 기술된 방식을 지적할 수 있고), 내용 해석에 대한 의견이 다를 수 있다(근거를 제시하여 의견을 제시할 수 있다.) 문제제기는 팀 단위로만 가능하고, 그에 따른 보상도 팀 단위로 들어간다.

Inevitably, some students will disagree with your selection of a best answer on a RAT question. They will do so on one of two bases: the question was written in such as way that they were confused or they feel you made an error in your interpretation of the content. The appeals process (Michaelsen 2008, p.24) provides the opportunity, preferably while they are still in class, to either re-write a question that they feel was poorly written or articulate, in writing, why they feel their answer was better, using references if appropriate. Accept appeals from a team only; award credit to the appealing team(s) only and to the individual scores of the members of those teams.

장점

The appeals process provides a number of benefits. 

• One is that it motivates students to do a focused re-study of the exact material that gave them the most trouble. 
• Another is that, the process of trying to put together a successful appeal requires to think deeply about both the specific ideas and the overall context within which they reside. 
• Finally, students can often re-write your questions so that indeed they are better!

학생들의 상호 동료평가 활용

Tip 9: Peer evaluation is a challenge to get going, but it can enhance the accountability of the process

다음과 같은 장점이 있음.

There are several ways to set up a peer evaluation process for the course, and it may take some trial and error to find the one that fits well with your institution or course's culture (Levine RE 2008, Chapter 9). There are, however, numerous benefits from putting forth the effort. 

• One of the most important is that, when you use peer evaluations, students are accountable to the members of their team. 
• Another is that a well-designed peer evaluation process enables students to learn how to give constructive feedback to one another and to gratefully receive constructive feedback from peers – an invaluable competency for future practice.

사전 준비에 대해서 명확히 하기

Tip 10: Be clear and focused with the advanced preparation

TBL에 대해서 교수자가 사전에 공부할 부분을 정해줘서 학생이 스스로 무엇을 공부해야 하는가 알아낼 기회를 뺏는다는 비판이 있지만, TBL에서 사전 준비란 RAP의 '개인별 파트'에 대해서만 도움이 되도록 하는 것이다.

A criticism of TBL is that the instructor identifies the learning needs for the students, thereby robbing them of the opportunity to explore the potential domain of the content and make some judgments about what they need to know. Based on past experience, when you are specific about what you want them to master before a TBL module, including posting action-oriented objectives such as ‘Be able to articulate how dopamine affects sodium channels at the receptor level,’ you invite them to go beyond doing the minimum of preparation since that will only help for the individual part of the RAP. They learn quickly that for their team to be really successful in the group work, they must master the advance assignment assiduously and devote additional effort to exploring the content domain. Tying the TBL objectives to the course objectives is essential.

팀 구성의 요령

Tip 11: Create the teams thoughtfully

세 가지 원칙이 있다.

We have three principles to guide the process of getting a class into teams: 

• (1) make the process transparent so all students know how they ended up in a particular team, even if the process is totally random; 
• (2) distribute what you define as ‘resources’ for a team as evenly as possible, for instance, a beginning class of medical students might have several students who have advanced degrees in one of the basic medical sciences, so you want to assign them to different teams; and 
• (3) strive for the teams to have a diverse composition, i.e. gender balance, rural or urban backgrounds, science/nonscience majors. Letting a class know that teams that have diversity within, however defined, will have unique strengths to draw upon in the challenging modules ahead.

작은 예산으로 효과를 내기

Tip 12: Several low-budget ‘props’ facilitate the implementation of a good module

One does not need to spend several thousands of Euros for the latest audience response system or any high-definition technology to get a well-constructed TBL module to work. We recommend using IFAT™ response forms for the group readiness assurance because students will hover over the scratch-off card, talk with each other, make eye contact, and be passionate about whether or not the correct answer is going to emerge. They receive immediate feedback, let one another know things like ‘You were right! Next time make us listen to you!’ Prepare folders for each team, color code the components to make the sequence of activities clear, collect everything so that you do not have to start ‘de nova’ every year – a good module is a treasure. Buy or build flagpoles to demarcate the position of teams; laminate the lettered cards for simultaneous responses. Require students to stand and face the class when speaking – you will not need a roving microphone once they learn to be quiet when someone is speaking.

결론 Conclusions

We are grateful to have been invited to provide these 12 TIPS. Over the past few years, we have provided many faculty development workshops and consultations, around the world, to introduce medical educators to TBL and assist them with its implementation in a variety of settings. In most cases, TBL has produced a positive transformation of the classroom experience for both the students and the instructor. Sometimes, however, we hear comments from faculty such as: ‘I tried it a few times, but gave up because the students didn’t like it,’ or ‘Does one have to use all the components? The GRAT sounds like a waste of time.’ Unfortunately, whenever we ask about the details of a less-than-successful attempt, we almost always learn that one or more of the components had been omitted or altered substantially. The strategy has been well tested and works, but works best when all of the components are included in the design and implementation.

Med Teach. 2010;32(2):118-22. doi: 10.3109/01421590903548562.

Twelve tips for doing effective Team-Based Learning (TBL).

의대를 다니는 동안 마음이 오히려 차가워지는가? (Academic Medicine, 2008)

Is There Hardening of the Heart During Medical School?

Bruce W. Newton, PhD, Laurie Barber, MD, James Clardy, MD, Elton Cleveland, MD, and Patricia O’Sullivan, EdD

의료 전문직업성의 중요성과 구성. 교육을 통해서 길러줘야 함에도 의과대학 기간동안 특정 부분에 대해서는 안좋은 영향만 주고 있다는 연구들이 있다. 냉소주의가 높아지고, 윤리적, 도덕적 발달이 저해된다.

Medical professionalism is essential for maintaining the integrity of the profession, and it includes demonstrating compassion, caring, and a willingness to put the concerns of patient and society above one's own. Medical education should promote the development of these professional qualities. However, studies have shown that medical school can often have a detrimental effect on certain aspects of students’ professional growth. Negative characteristics such as cynicism may increase, and ethical and moral development can be stunted.1–3

공감능력은 반드시 키워야 할 중요한 전문직으로서의 자질이다. 환자의 만족도를 높이고 치료를 더 잘 따르게 한다. 의사의 공감능력은 환자가 인지하는 치료자로서의 능력에도 영향을 준다. 진료기술이 뛰어나도 이러한 환자만족, 치료에 대한 순응, 의사의 공감능력에 대한 중요성을 모르면 환자는 '비효과적'이라고 느낄 수 있다.

Empathy is one of the most highly desirable professional traits that medical education should promote, because empathic communication skills promote patient satisfaction and adherence to treatment plans while decreasing the likelihood of malpractice suits.4,5 Patients view physicians who possess the quality of emotional empathy as being better caregivers. A physician may possess competent diagnostic skills, yet be considered by patients as “ineffective” because the physician misses the link between patient satisfaction, adherence to medical instructions, and physician empathy.

공감능력은 두 가지로 분류된다. 간접적(vicarious), 상상적(? imaginative)이다. 

Sociologists and psychologists have divided the concept of empathy into two main definitions or types: vicarious and imaginative. 

• Vicarious empathy 다른 사람의 감정적 경험을 인식하는 것. is “an individual's vicarious emotional response to perceived emotional experiences of others” and 
• imaginative empathy 다른 사람의 입장에서 그 사람의 생각, 감정, 행동을 정확히 이해하고 예측하는 것 is “an individual's ability to imaginatively take the role of another so as to understand and accurately predict that person's thoughts, feelings and actions.6 

첫 번째는 정서적 반응이다(gut reaction), 두 번째는 인지적 공감(cognitive empathy)이다.

The first definition reflects an innate emotional response, that is, a “gut reaction,” and is equivalent to the “empathic concern” described by Davis 7; the second definition refers to “cognitive” empathy and reflects a learned ability to imagine and intellectualize.7

여러 척도들이 '인지적 공감'을 측정한다. 기존의 연구에서 의과대학생의 인지적 공감은 변화가 없다고 나온 것, 올라간다고 나온 것, 내려간다고 나온 것 등 다양하다. 우리의 이전 단면연구에서 보면 의과대학기간동안 공감이 감소한 것을 확인하였다. 

Many scales that measure empathy are investigating cognitive empathy of individuals to “role-play.”8 Previous data concerning medical students’ cognitive empathy are conflicting, indicating either no changes, decreases, or increases in empathy during undergraduate medical training.9–13 In our previous cross-sectional study, we observed a decline in vicarious empathy during medical school.14 There are no known longitudinal studies of vicarious empathy. In the current study, we examined the longitudinal effect of medical education on vicarious empathy.

Methods

Setting and sample.

We gathered this study's data from a single South-Central U.S. medical school: The University of Arkansas for Medical Sciences. Starting in the 1997, 1998, 1999, and 2000 academic years, the students in the graduating classes of 2001, 2002, 2003, and 2004 completed a survey at the beginning of their freshman (M1), sophomore (M2), junior (M3), and senior (M4) years. With approval from our local institutional review board in 2005, we contacted the 535 graduates and asked whether they would allow their data to be used in a research study. A total of 419 (78.3%) students agreed, 1 student refused, and 115 students did not reply.

Survey instrument.

The survey instrument was the Balanced Emotional Empathy Scale (BEES), a well-established measure of the vicarious emotional qualities of empathy that examines the emotional “primitive” level of interpersonal interactions.8,16 The BEES coefficient alpha is 0.87. The BEES consists of 30 positively or negatively worded items (15 items in each category) that measure responses to fictional situations and particular life events. Because the BEES is gender sensitive, with men scoring lower than women, the students were asked to report their gender. Using the students’ responses to the BEES, we analyzed the changes in the students’ vicarious empathy by gender across their first three years of medical school.

Specialty choice.

We classified each student's specialty choice on the basis of his or her residency match at the time of graduation. Specialty choice was divided into two categories, five core specialties (i.e., internal medicine, family medicine, pediatrics, obstetrics–gynecology, and psychiatry) and noncore specialties (all other choices, for instance, surgery, pathology, and radiology).

Timing of test administration.

Students completed the questionnaire during registration for each academic year. Therefore, the M1 classes had no medical training, providing a baseline empathy score.

Analysis of data.

Scores for each student were calculated according to BEES instructions.16 Descriptive statistics were calculated by class, gender, and choice of specialty. As stated earlier, we conducted separate analyses for men and women, because the BEES is gender specific. For each year, we conducted a two-factor repeated-measures ANOVA, using the general linear model procedure from SPSS, with the significance level set at P < .05. The between-subjects factor was specialty choice, and the within-subjects factor was time. Significant effects were followed by post hoc tests. We also performed single-sample tests of means to compare our respondents with those in the normed sample for the BEES.

Discussion

공감능력이 의학교육에 의해서 영향을 받는다는 결과. 오히려 남학생은 일반 사람들보다 더 높았다.

The results of this study suggest that student empathy is affected by medical education. Our study supports the findings of Coulehan and Williams,18 who described deleterious changes in various humanistic qualities as medical students became “immunized” against these values after their matriculation into medical school. In the freshman year, the similar M1 BEES scores for all four classes indicate that the matriculates studied were drawn from a homogenous population, and that the undergraduate education of the matriculates, predominated by basic science courses, did not decrease vicarious empathy below the established norm. Indeed, the entering male medical students had BEES scores that were significantly higher than the norm. For both the men and the women, the M1 BEES scores were not significantly different between core and noncore groups.

의과대학 1학년이 지나며 크게 감소함. 1학년을 지나면서 미디어에서 보여지는 것과 실제 모습이 차이가 있다는 것을 인지하게 되었을 것이다. 1학년들은 자신들이 애들 대접을 받는다고 느낄 것이다. 이에 대한 대응적 행동으로 냉소주의가 높아지고, VE가 감소한 것으로 본다. 

For all the students studied, the significant decrease in vicarious empathy that occurred after completing the freshman year of medical school may have resulted from a high degree of student stress and anxiety caused by the students’ competitiveness and desire to overachieve on examinations. Additional stressful factors may include the media's presentation of doctors as heroes, which helps create a skewed image of the ideal physician for entering freshmen medical students. As the students progressed through their freshman year, they probably realized there is a mismatch between the media representation and reality.19 Additionally, the freshmen likely considered themselves to be in a hostile educational environment that treated them like children.20 All of these stressors induce self-preservation and coping behaviors. In our view, some of these behaviors, expressed as increased cynicism 2 and, ostensibly, decreased vicarious empathy, were used by the students to adjust to the stresses and internal conflicts associated with medical education.

2학년 기간에는 BEES 점수가 유지된다. 학생들이 적응한 것으로 보인다. 3학년 임상실습을 돌며 크게 다시 한 번 감소하는데, 이는 임상실습 첫 해에 대해 학생들이 느끼는 부담이 작용한 것으로 보인다. 특히 환자를 보는 경험이 있는 동안 크게 감소한다. 최신기술을 적용하는 것이 치료에 있어서 중요하게 강조되는 3차병원에서는 환자에 대한 공감을 형성하기가 쉽지 않다. 

The sophomore year, which is an additional year of basic science courses similar to the freshman experience, maintained stable BEES scores. This may be explained by the students’ acclimating to the academic rigors of basic science courses. The junior clinical year, where all students participate in standard rotations, produced another dramatic drop in BEES scores for both the men and the women. This drop substantiates findings from the literature that suggest that the first clinical year of medical school is demanding and challenging.21 The large drop in M3 students’ vicarious empathy occurred while the students were seeing patients they had, presumably, looked forward to helping. It is interesting to note that a recent study showed that a similar drop in empathy occurred after dental students began seeing patients.22 In university tertiary care centers, empathy is hard to direct toward the challenging patients when the treatment emphasis is on technology.23 Immense cultural differences between physicians-in-training and patients may also make it more difficult to achieve adequate empathy. Physicians who are role models should work more closely with medical students to develop an empathic relationship with such patients.24 Our own findings suggest that this could be particularly important for women who enter a noncore specialty, because their drop in vicarious empathy scores more closely emulated the naturally lower empathy scores of the men than the scores of their female colleagues who selected a core specialty.

기존 연구에서 공감능력을 바탕으로 한 환자-의사 관계가 프로페셔널리즘의 중요한 측면 중 하나이며, 프로페셔널리즘과 환자-의사 관계는 병원 내 롤모델로 삼을 만한 의사들에 의해서 가장 잘 보여지게 된다고 나타난다. 다른 연구는 임상 롤모델이 계속 부족해왔음을 지적하는데, 이러한 긍정적인 롤모델이 부족한 것이 공감점수의 감소에 영향을 주었을 것이다.

Studies show that clinicians consider an empathic physician–patient relationship as one of the most important aspects of professionalism 5 and that professionalism and physician–patient interactions are best demonstrated by clinician role models at the bedside in hospitals or,25 as suggested by Benbassat and Baumal,24 in primary and chronic care clinics and/or hospice facilities. Other studies reveal a chronic lack of clinical role models,26 and perhaps a lack of positive role models could have contributed to the decline in empathy scores we observed after the completion of the first clinical year of undergraduate medical education.

복잡한 치료 요법이나 임상 롤모델의 부족함 외에도, 갓 의사가 된 사람들이나 학생들은 그들이 멘토나 롤모델에 의해서 남용(abuse)되고 있다고 느낀다는 점이 보고되고 있다. 이러한 학생에 대한 남용은 "traumatic deidealization"으로 불리곤 하는데, 이 역시 영향을 주었을 것이다. 누적되는 피로와 로테이션에 의해 환자에 대한 연속성이 떨어지는 것도 상황을 악화시켰을 수 있다.

In addition to complicated treatment regimes and a lack of clinical role models, literature shows that juniors, as student/physicians (i.e., individuals that are learning to assume an authoritative role in providing care while simultaneously being subservient to their mentors), perceive themselves as being abused by their mentor/role models.1,21,27 This student abuse has been termed “traumatic deidealization” by Kay 19 and is another factor possibly contributing to the declines in empathy. Other potential factors include fatigue and a lack of patient continuity that is exacerbated by starting a new clinical rotation every four to five weeks.

'의도한' 교육과정은 '비공식적' '숨은' 교육과정에 의해서 학생들의 기대에 못미치게 된다. 비공식 교육과정과 숨은 교육과정의 정의. 

The “intended” medical curriculum, which is the formally offered and endorsed curriculum, often falls short of student expectations because of the “informal” and the “hidden” curricula.28 

• The informal curriculum, as defined by Hafferty,28 “is an unscripted, predominantly ad hoc, highly interpersonal form of teaching that takes place among and between faculty and students” (e.g., in the hallway, lounge, or on-call room), whereas 
• the hidden curriculum “is a set of influences that function at the level of organizational structure and culture” (e.g., advertising the amount of National Institutes of Health dollars the institution garners or espousing the need for basic science rather than humanities courses when applying to medical school). 

이 두 가지가 바람직한 프로페셔널리즘을 은밀히 저하시키는 역할을 하게 되는데, 이는 학교의 정책이나 학교가 자원을 투자하는 부분과 '실제로 중요한 것'이 갈등을 일으키는 것을 학생들이 쉽게 파악하기 때문이다. 대신 비공식, 숨은 교육과정에서 강조하는 것은 의과대학의 사회적 의무와 학생의 공감 및 기타 직업전문성적 특성을 배양을 도와줄 수 있는 롤모델의 양성이어야 한다.

These latter two curricula contribute to insidious declines in desirable professional traits, because students easily recognize that campus policies and resource allocations and “what's really important to learn” often conflict with the stated institutional educational mission.1,28 Instead, the mandate that should be implicit within the informal and hidden curricula should be the societal obligation of medical schools to provide and/or train better physician role models who can aid students in overcoming losses in empathy and other professional characteristics.26

가장 흔하게 제시되는 대안은 공감적, 인문학적 교과목을 넣는 것이다. 그러나 많은 연구에서 이러한 식의 대안은 한시적이고 큰 효과가 있지 않으며, 어떤 사람들은 공감능력이 가르쳐서 기르기에 어려운 것이라고 느끼기도 한다. 전통적인 4년제 의과대학과 달리 최근 일부 6년제 학교 또는 PBL을 기반으로 하는 학교에서는 인문사회의학이나 공감에 대한 내용을 교육과정에 넣기가 더 용이하다는 점이 보고된 바 있다.  

The most frequently offered solutions to ameliorate the loss of professionalism at the undergraduate medical level involve teaching students to be empathic or “humanitarian” via courses that emphasize empathic communication skills.9,12,29 Evidence suggests that in most instances the gains are modest and temporary,10,30 and some feel that empathy and compassion are increasingly difficult to teach as an individual matures.20 In contrast to traditional four-year schools, several recent studies have shown that six-year medical schools, or schools with a problem-based learning curriculum, have greater opportunities to integrate humanism and empathy into their curricula.31,32 

이들 '비전통적'학교의 초기 결과보고는 긍정적이지만, 많은 사람들이 의대생이 가지고 있는 해로운 무력함(detrimental inertia)이 윤리, 도덕, 공감, 봉사지향적 태도에 의해서 가속화되고 이것이 얼마 안되는 교육을 더 한다고 극복되지 않는다고 느낀다. 일부 전공의 프로그램은 이러한 사실을 깨닫고 집중워크숍을 통해서 가르쳐보고자 하나, 그러한 교육에도 불구하고 인턴 시작 후 다섯 달만 지나면 이상주의나 공감은 감소하고 냉소주의는 올라간다는 것을 보고한 바 있다. 

Despite some encouraging initial reports from several of these nontraditional schools, a large cadre feels that medical students possess a detrimental inertia, fueled by a frank decline in ethics, morals, empathy, and service-oriented attitudes, that cannot be overcome by a concurrent limited exposure to the humanities, especially when presented in an already crowded four-year curriculum.2,9,33,34 Some residency programs, aware of the reduction in humanitarian traits in medical graduates, have developed intensive workshops devoted to teaching and enhancing physician–patient interactions, yet studies show that even with such courses, idealism and empathy decrease and cynicism increases in as little as five months after the start of the intern year.35–37 

We maintain that the attempted maintenance of professionalism should be part of the medical school curriculum that is combined with the reinstatement of effective bedside teaching. However, our study findings suggest that the loss of innate, vicarious empathy could make it difficult to effectively teach medical students, interns, and residents to consistently role-play empathic concern.

Acad Med. 2008 Mar;83(3):244-9. doi: 10.1097/ACM.0b013e3181637837.

Is there hardening of the heart during medical school?

의과대학 신설: 새로운 지역으로의 교육 확장

Developing a medical school: Expansion of medical student capacity in new locations: AMEE Guide No. 55

DAVID SNADDEN1, JOANNA BATES1, PHILIP BURNS2, OSCAR CASIRO1, RICHARD HAYS3, DAN HUNT4 & ANGELA TOWLE1

1University of British Columbia, Canada, 2University of Manchester, UK, 3Bond University, Australia, 4Association of American Medical Colleges, Washington DC, USA

새로운 지역으로의 의과대학 확장

Expansion of medical student capacity in new locations

여기서 '기능적으로 분리된 캠퍼스'란 지역적으로 구분되는 장소에서 진행되는 의학교육프로그램으로서, 이러한 지리적 거리때문에 학생들을 위해서 시니어리더십, 교수, 행정, 학생지원 등을 제공해야하는 캠퍼스를 의미한다. 캠퍼스는 기존의 의과대학이나 의과대학, 의과대학을 보유하지 않은 지역 대학, 지역의 병원 또는 보건기관과 협력관계에 있을 수 있다. 이러한 캠퍼스는 전임상실습, 임상실습, 혹은 둘 다를 위한 것일 수 있으며, 모(母)의과대학과 서로 다른 설립이념을 가지고 있을 수도 있다.

By a functionally separate campus, we mean a geographically distinct medical education program where, by reason of the amount of time spent there by students or because of geographical distance, the campus must provide support services for students including its own senior leadership, faculty, administration and student services. The campus may be affiliated with an established university and medical school, with a local university that does not operate a medical school, or a local hospital or health facility. Such a campus maybe pre-clinical, clinical or both and may be philosophically distinct from its parent medical school, for example it may emphasize rural training.

Background

21세기 초반 많은 국가에서 의사 부족을 예견하고 의과대학 확장이 빠르게 이루어졌다. 확장을 하게 된 것은 더 많은 의사가 필요했기 때문만이 아니라 의사인력의 분포가 지역적으로 불균형하다는 것에도 있었다.

In the first few years of the twenty-first century, there has been recognition that many nations will face doctor shortages. This has led to the rapid expansion of medical school enrollment in several countries. Expansion plans have been driven not only by the need for more doctors, but also by the uneven distribution of the current medical workforce both geographically and by discipline.

의과대학의 지역 캠퍼스는 새로운 현상이 아니다.

Regional campuses are not a new phenomenon.

• In the USA, the first wave of regional campuses was established in the 1970s in response to the expansion of medical school enrollment. The Association of American Medical Colleges (AAMC) reported in 2006 that 20% of the medical colleges in the USA had regional campuses (Mallon et al. 2006). These regional campuses deliver pre-clinical and clinical education or one of these elements alone. For example, the distributed education program at the University of Washington that started in 1971 (Ramsey et al. 2001) known as the WWAMI program for the five states that it serves, has seven pre-clinical campuses where students in their home states of Washington, Wyoming, Alaska, Montana, and Idaho receive the first year of pre-clinical education. Other regional campuses offer the clinical component of the education program, such as the 30 US regional campuses described by the American Colleges of Medicine (Mallon et al. 2006). 
• In Australia, regional campuses were developed in collaboration with existing medical schools to address the shortage of rural health care providers (Prideaux et al. 2001). Some of these went on to develop as independent medical schools (e.g. James Cook University). 
• In the UK, as in many countries, medical school provision is government controlled and dependent on predictions of future workforce needs. Expansions were made in medical student numbers in the late 1990s and early 2000s based on predicted UK workforce needs (Howe et al. 2004) in a time of economic growth and prosperity. Planning for workforce development and government controlled increase in student numbers has been the main driving factor for the development of several new medical schools and regional campuses (Medical Workforce Standing Advisory Committee 1997). 
• In Canada, the first fully distributed campuses – ones where the majority of the education occurs on the distributed site – were developed by the University of British Columbia (UBC) in partnership with the Universities of Victoria and Northern British Columbia in 2004 (Snadden & Bates 2005). This was paralleled by the expansion of distributed campuses in the USA (Rackleff et al. 2007) and followed by the creation of similar distributed sites elsewhere in Canada. At least, one new medical school was created as a fully distributed and community engaged school (Strasser & Strasser 2007), thus highlighting that as medical schools expand or are created there is no one model to suit all contexts, to encourage diversity or to support the varied social responsibility missions of different schools. 

왜 지방캠퍼스인가? Why regional campuses?

약간의 학생 수 증가로도 상당히 많은 복잡한 문제들이 생겨난다. 그러나 여러 문제들에도 불구하고 지방 캠퍼스의 장점들이 있다.

Expansion brings a number of challenges that can stress a medical school, though the degree of complexity is related to the number of students involved. Meticulous planning is important for any increase in student numbers, but the complexities of increasing by a small number of students rise exponentially if a school grows by 50–100 students, or doubles their class size and engages new partners at new regional sites (Bunton et al. 2008). Despite the increased complexity, there are advantages to developing a regional campus (Mallon et al. 2006).

Regional campuses

• Give medical schools access to a larger patient base and expanded numbers of clinical teachers.
• Allow a focus on disciplines such as primary care that may be more difficult to develop on the main campus.
• May attract a different cohort of applicants to the medical school.
• Can offer a different type of education such as a more clinically focused or more integrated program.
• Benefit regional hospitals by enhancing their standing as academic centers, improving recruitment and having a positive impact on quality of care (Hanlon et al. 2010).
• Benefit the local community by increasing physician recruitment, increasing civic pride and providing some economic benefits associated with an increase in social capital (Lovato et al. 2009).

단계1 : 기획 

The first steps – planning

새로운 지역 물색하기

Identification of a new campus site

새로운 캠퍼스의 기능에 달려 있다.

Identification of a new site will depend on the nature of the need for a functionally separate campus.

학생 숫자가 절대적으로 얼마나 증가하느냐와 더불의 의사의 분포에도 관심을 기울여야 한다(사회적 책무 미션)

Workforce planning considerations may not only require an absolute increase in student numbers but also attention to physician distribution. These considerations lead to more specific considerations for the location of regional campuses, as the criteria go beyond the suitability of the site to deliver education, but the appropriateness for delivery of this additional social responsibility mission. 

In Canada, the Northern Medical Program (NMP) and Island Medical Program (IMP) in British Columbia, and the Northern Ontario School of Medicine were a result of both insufficient supply of graduating physicians and an uneven distribution of practicing physicians between urban and rural regions.

작은 규모의 지역사회가 핵심 임상경험을 하기에 더 적합할 수도 있다.(평가인증 기준에도 더 잘 맞을 수 있다)

It may be a challenge in smaller communities to provide students with an appropriate set of core clinical experiences (which may be influenced by accreditation standards). Two examples from Canada illustrate creative solutions to this issue. 

파트너십 강화 Developing partnerships

Once the regional campus site has been determined, the development of a working collaboration between the parent university and partner institutions, whether they be university or health service or both, is critical. Features of partnership development include:

• 공통의 이해관계 찾기 Finding common ground
• 윈-윈 관계 형성 Creating a win–win relationship
• 공동의 비전, 미션, 목표 수립 Building common vision, mission and goals
• 주인의식 고취 Promoting ownership
• 공동의 추진원칙 수립 Acting on a set of agreed guiding principles.

Written commitment to a common mission, a set of common goals, and a clear set of operating principles at this point will prove invaluable when discussions become difficult. 

의료취약지에서는 진료의 패턴 차이가 커서 갈등의 요인이 될 수도 있다. 예컨대 취약지에서는 generalist가 해야 하는 업무의 폭이 더 넓다. 

Indeed, if the area is underserved, there may be variations of patterns of clinical practice that can create conflict. For example, the scope of practice of generalists may be much broader in an underserved regional setting that in a university-based tertiary care setting. These differences are best articulated clearly and addressed before planning gets underway.

외부적 전략도 필요하다.

External strategies to support a common purpose may include public meetings, social events, regular media contact and regular meetings with the partners, ideally with independent facilitators to get through issues where there is incomplete agreement. Internally, an intense change management strategy of engaging faculty and staff across sites becomes essential.

지원체계 구축 Building support

지역사회의 의사들과 구성원들과의 협조를 구축해나가야 한다.

As relationships are built between academic institutions, engagement with the health services sector and the community must begin. Buy-in of the local physicians is essential but may not have occurred if the partnerships have resulted from inter-institutional agreements that have not involved the physician community. The regional academic partner may be unprepared for the degree to which physicians expect to be engaged: the parameters for normal community consultation by a university may be woefully inadequate for the health care delivery community. Remuneration levels for physician executives may be shocking to the regional university and disruptive to their culture. The process of engagement is delicate: identifying and engaging the local leaders is key. Finding the supporters and early adopters will provide an initial base to build on.

취약지에서 의사들은 환자도 보고 교육도 해야 하는 것에 부담을 느낄 수 있다. 일부는 심지어 '학계'를 떠나기 위해서 왔을 수도 있다. 그러나 이들조차도 동료들이 학생들과 즐겁게 지내는 것을 본다면 무슨 일이 벌어지나 관심을 갖게 될 것이다. 

In underserved communities, physicians may feel overburdened by both looking after patients and teaching, and some may also have moved to that community specifically to escape “academia.” Most physicians, even the busy ones and those who abhor the trappings of academia, will come around when they see their colleagues’ pleasure in working with learners, so it is important to acknowledge these two statements of resistance but not be discouraged in moving forward.

어떤 사람들의 새로운 의과대학의 교육 미션과는 거의 관계도 없는 오랜 고충을 해결하기 위해서 대학, 보건당국, 병원, 정부와 협상하기 위한 수단으로 지방 캠퍼스를 생각할 수도 있다.

Furthermore, some groups may see the prospect of a regional campus as an opportunity to bargain with the university, health authority, hospital, or government to address long standing grievances that have little or nothing to do with the medical school's teaching mission. 

더 넓은 지역사회의 지원 Support in the broader community

Support in the broader community is critical to success. It is important to engage local community leaders and politicians and to find ways to ensure the local population are informed and have a chance to become engaged in the new development. A mixture of meetings with local leaders from outside of medicine, local politicians, media releases, and public events are ways of engaging.

지역사회의 비현실적인 바람을 적절히 관리하는 것이 필요하다. (경제적인 기대 등) 졸업생이 실제 진료를 하기까지는 오랜 시간이 걸릴 것이고, 오히려 실망하게 될 수도 있다.

Finally, volunteers from the community can be very helpful in terms of developing simulated and standardized patient banks and many feel they are contributing to a broader social mission. It is also important to manage unrealistic expectations from the community. There will be some economic gains with the addition of a regional campus. However, recruitment of graduates into practice will take many years, and overly enthusiastic press releases can end up with later disappointments.

초기 리더십 Early leadership

현장에서 뛰는 사람이 필요하다. 의학교육에 대해서 별로 아는 것이 없고 경험이 없을 수도 있지만, 여러 사람들을 만나고 다닐 필요가 있다.

It is likely that in the initial stages the regional campus leadership will consist of individuals who are doing this ‘off the side of their desk’. They may know little or nothing about medical education in general, have been educated in a different paradigm or country, and have little or no experience of the parent program in particular. It is important early on to identify temporary or interim leadership who can fully engage with counterparts at the main campus until a full search process can take place.

그 지역의 사람들을 참여시켜야 한다.

Involvement of local people in the selection process is key. You are looking for people who are collaborative and good communicators; credibility with the local community trumps experience in medical education.

기획 단계 Planning process

리더십이 구축되고 동의가 구해지면 구조와 절차를 기획해야 한다.

Once leadership is in place and initial agreements are set, a planning structure and process is required. This should be collaborative and formal, ensuring that progress is made toward formal recommendations and approvals of processes.

기획 단계에서 이런 것들이 고려되어야 한다. 입학에 대해서는 개교 2년 전에는 확정되어야 한다.

Planning must include necessary infrastructure including new buildings, technology requirements, and ongoing funding. Policy-intense issues such as the approval processes for budget allocations, student support and fees, allocation of bursaries across sites, and processes for faculty appointments should be sorted out at this time. Although we focus on admissions in a later section, admission processes must be finalized 2 years before the entry of the first students, and often serve as a focus for the partners in creating and agreeing on concrete processes, based on common missions and goals.

교육과정 모델이 만들어지면 기초의학자, 튜터, 임상실습강사 등을 다양한 분야에서 모집하기 위한 구체적인 내용을 결정해야 한다.

Once the curriculum model is established, a curriculum mapping exercise can provide the detailed requirements for basic scientists, tutors and clinical preceptors from different disciplines. These needs can be matched against existing resources and future hiring plans

거버넌스와 재정 Governance and finances

The more successful models have a clear, shared vision and mission, clear lines of communication and accountability, greater reliance on local decision-making to solve local implementation challenges, and clear guidance for resolving differences between partners.

인증기준에도 관심을 두어야 한다.

In designing these, it is essential to keep a close eye on accreditation requirements. 

재정 구조를 확실히 하기 위해서 어떻게 자금이 유입되고 재정적 자율권은 어느 정도 되는가를 명확히 해야 한다. 나중에 변화할 부분을 위해서 유연성이 있어야 한다. 초창기에는 명확히 기술되어 서로 동의하여 서명한 동의서가 불필요해 보일지 모르겠지만, 시간이 지나고 인력이 바뀌고 기억이 흐릿해지고 우선순위가 뒤섞이는 경우를 대비해서 반드시 필요하다.

In addition be clear about the financial structures, how money will flow and the degree of financial autonomy on the regional campuses. In the early days of building a new campus, the financial formulae used will be best guesses, so it will be important for the campus to have flexibility in how it uses its resources in order to solve unexpected problems. Clear budgeting processes, committee membership, and conflict resolution mechanisms must be agreed and written down in signed inter-institutional agreements. While these may seem unnecessary during the initial enthusiasm that goes with new developments they will prove essential in time as personnel change, memories fade and different priorities appear on the political and institutional landscapes. It is easier to write and negotiate them at the beginning than several years into operations.

평가인증 Accreditation

국가 기관에 대한 평가인증은 지방 캠퍼스가 성공하기 위해서 반드시 필요하다. 매인 캠퍼스에 집중될 것이지만, 분명 평가단은 지방 캠퍼스도 방문하고 싶어할 것이다. 

Accreditation by national agencies is essential to the success of the regional campus and requires careful thought from the initiation of the project. While the main focus of accreditation may be a variation of the program provided by the main campus, accreditation agencies will almost certainly want to visit the regional campus to ensure that there are adequate resources, a positive student experience and appropriate graduate outcomes.

기획이 종료되면 - 착수 

Once the planning is over – getting going

시설 개발 Facility development

딜레이를 대비하고, Plan B를 가지고 있어야 한다.

One of the truisms of developing new ventures is that the buildings and other infrastructure are often not completed according to the original timeframes. Developing a regional campus is a complex task, as the campus, although smaller than the main base, must have everything necessary to deliver a complex medical program. There are therefore two essential elements to the planning: allow for delays in completion; and have a ‘Plan B’.

딜레이를 인정하는 것은 일정 부분 회의적인 자세가 필요하지만, 상당히 어렵다. 

Allowing for delays in completion perhaps requires a degree of educated pessimism, but is quite difficult.

사용하지 않고 있는 지역의 시설을 이용하는 대안(plan b)를 준비해야 한다.

All of these commonly disrupt plans. Hence having a Plan B is essential – offices for faculty and other staff, teaching space, and basic IT infrastructure can be created by taking over local facilities that may be idle. 

지역 파트너들과의 관계 Relationships with local partners

Most regional campuses involve collaborative partnerships between the central university and local organizations, because that is usually the most effective and efficient way of proceeding.

시작 시점의 파트너십은 점점 지연되면서 시험대에 오를 것이다.

The initial planning (second section), outlines initial engagement of local partners to get the plan to a supported, implementable stage. However, this is really only the beginning of the partnerships, which may be tested by delays in construction, changes in policy or changes in the objectives of the partners. As at the beginning, this requires excellent communication strategies both externally and internally.

메인 캠퍼스와의 관계 Relationships with the main campus

지역 캠퍼스는 그 지역의 건강요구에 부합하는 그 자체의 미션과 에토스를 만들어야 한다.

An important variant of maintaining relationships is the relationship between the regional and main campuses. The regional campus inevitably develops an ethos and mission of its own, often based on local workforce development and meeting local health care needs.

메인 캠퍼스와 긴장이 있을 것이다. 그럴 때는 큰 그림을 봐야 한다. 

It is almost certain that there will be tensions between the main and regional campuses. At times, all partners have to look upwards toward the lofty vision and goals initially developed – the bigger picture – rather than looking too closely at the smaller steps along the way. 

교수와 직원 모집 Recruitment of faculty and other staff

특정 사람들에게는 매력적일 수 있다(self-reliant, higher risk taking tendencies). 

Recruitment of faculty to new initiatives in distant locations can be a real challenge. It may be that these positions are attractive to particular types of individuals – perhaps those with more self-reliant, higher risk taking tendencies. Traditional academic research careers are difficult to nurture in regional campuses, although there may be opportunities to build research and development around local industry strengths, local population and health care needs, and the process of developing and evaluating innovative educational approaches.

어떤 지역 리더십을 선택하느냐는 중요한 결정이다.

The choice of local leadership is a crucial decision. The leader must share the vision, be strategically flexible, be innovative, be prepared to work with all partners to push the agenda along the planned path, and be willing to adapt development in response to inevitable challenges.

교수 개발 Faculty development

Most new ventures commence with a small core ‘pioneer staff’ comprising some external people but in general a majority of local recruits. This means that faculty development is a major early priority.

가장 연구로 성공할만한 영역이 어딘가를 찾아서 집중하는 것이 좋은 초기전략일 수 있다.

An important early strategy is to determine where the best chances of research success lie. There may be local health issues (e.g. rural or indigenous population health) that will attract funding because of the location and therefore be more successful than similar research development at the main campus.

학생 선발과 모집 Student admissions and recruitment

두 가지 방법이 있다.

Because admissions processes are formalized 2 years prior to the start of classes, issues related to student admissions need to be addressed early in the relationships between partners. The easiest path for large, established medical schools is to simply select more students from their competitive pool and allocate them to the regional campus. On the other hand, local partners may prefer a local community development option that increases access for local youth to medical careers. This approach may be part of a broader local community development agenda that aims to increase participation of local people in tertiary education. 

그러나 이 둘 중 어떤 것도, 혹은 두 개의 조합도 실패할 것이다.

Neither of these paths, nor indeed a combination of them, is necessarily wrong, but it is essential to resolve any differences early. It is likely that the local community development argument will have to be acknowledged if strong local support is to be maintained – the ‘grow our own doctors’ mission is not just a powerful goodwill strategy, but it also has been shown to be effective (Hays 2001; Veitch et al. 2006). 

학생이 의과대학의 교육을 감당할 수 있어야 한다. 잠재적 학생과의 명확한 의사소통이 필요하다. 

On the other hand, sustainability of the regional campus will depend partly on the ability of the students to achieve in a demanding medical education program. In most jurisdictions, admissions are the responsibility of the faculty members, and admissions processes must be approved and defensible by the parent university senate. Development of policies and procedures that ensure potential regional campus students with high academic potential can be admitted can prove to be complex and a focus for conflict and misunderstanding. Clear communication with potential students is paramount.

교육과정 Curriculum issues

Developing regional campuses poses significant educational challenges above and beyond the challenges of establishing infrastructure and recruiting faculty. A major question to address is: to what extent is the curriculum able to be delivered at the new campus? This question has several dimensions, including 

• the mission of the new campus, 
• the content of the planned curriculum, 
• the match of clinical opportunities to the planned curriculum, and 
• the assessment of learning progress at all sites.

교육과정의 많은 부분은 면허시험기관(의 학습목표)에 의해서 결정된다.

In most jurisdictions the curriculum is largely determined by national licensing organizations, following a set of learning outcomes that define the capability of graduates from the relevant curriculum. Depending on the level of the program being delivered at the satellite campus, the challenges are different. 

실험 기자재 필요

For example, in early years the requirements are for the delivery of similar basic, behavioral and social science. This may include anatomy and physiology laboratory facilities. Lectures can be delivered across several sites by video or web-conferencing. 

균등한 수준의 임상경험을 쌓을 수 있어야 한다.

For more senior years, the challenge is to provide the appropriate clinical learning opportunities, which requires an appropriate range of patients, clinical problems and clinical investigation technology. While opportunities for clinical learning always vary, even within a single site, there needs to be equivalence of opportunities at all sites in order to prevent the perception that satellite campuses are less effective as learning environments. 

모든 캠퍼스의 학생들은 같은 기준과 방법에 의해서 평가받아야 한다.

Similarly, it is essential that students at all campuses be assessed in the same ways to the same standards.

두 캠퍼스에 대해 같은 평가를 하는 것이 손쉬운 방법이다.

In the US and Canada, assessment of students must be equivalent across the sites in order to meet accreditation requirements. The simplest way to start is to employ the same assessment methods and materials at the main and regional campuses. When all students take the same sets of exams, the results provide a rapid measure of comparability of academic performance across sites. Talent at the distributed site can be channeled toward contributions to the overall assessment pool (e.g. multiple choice questions) rather than to the development of a parallel assessment system. Effective, valid, and reliable assessment of student performance in small group activities (clinical skills, problem-based learning) by new tutors at the distributed site requires faculty development, peer support and mentorship. 

새로운 캠퍼스에서 활용하기 위한 교육, 평가 기술을 개발해야 함.

There are several approaches to developing the teaching and assessment skills of faculty at the new site. These can be used in combination, for example:

• send a small number of faculty to the main campus for short periods of training;
• send faculty from the main campus to the site for faculty development workshops, peer observation, support and mentorship;
• create opportunities for tutor development meetings and workshops by videoconference. 

한 번에 끝나는 과정이 아니다.

It is important to see this as a “train the trainer” ongoing process, not a one-time only endeavor. In a supportive, mutually respectful environment, experienced faculty at either site can observe and provide feedback to less-experienced tutors, enhancing the ability of new faculty to teach and assess students until a satisfactory level of proficiency has been reached. The initial learning curve is steep; lasting at least 2 years, fortunately this is when the body of new faculty is commonly enthusiastic, stable, and turnover is low.

지역 캠퍼스가 메인 캠퍼스보다 덜 만족스럽다는 인식은 대체로 틀린 것으로 드러났다.

In practice, the perception that regional campuses are less satisfactory than the main campuses is generally almost always proven to be incorrect. Curriculum relevance of clinical experience has been shown in the UK to be equivalent in distant community general hospitals to that in larger academic hospitals (Colquhoun et al. 2009). Student performance in centrally managed assessment has also been shown not to disadvantage learners at distant campuses (Worley et al. 2004; Bianchi et al. 2008). Indeed, many students and faculty believe that regional campuses often provide richer learning opportunities, with more general clinical case mix, lower student:patient ratios, and better opportunities for students to participate in patient care. There are a few students who may do better with the anonymity of the larger campus, but many of these students end up singing the praise of the regional sites. 

그럼에도 불구하고, 일부 메인 캠퍼스의 교수들은 지방 캠퍼스의 학생들이 뒤떨어질 것을 우려하고 있다. 공식적, 비공식적 의사소통 과정을 통해서 잘 설득해야 한다.

However, in spite of the evidence and the student acceptance, the faculty at the parent site may be anxious that the student outcomes at the regional campus will lower the outcomes overall of the medical school: it is important to articulate and address this issue through both formal and informal communications processes and local evidence.

인증기준에 어긋난 것이 아니라면 그 지역사회에 가장 잘 맞는 교육과정을 만드는 것이 좋다.

If the intention is to set up a campus that will in time be independent, thought needs given as to whether to import a curriculum from elsewhere or develop one locally. If accreditation processes allow, there is merit in developing a curriculum that truly matches the needs and challenges of the local community, rather than to use one that was developed in a separate context that may have less relevance. A modern curriculum that employs strategies to reduce the burden of factual information would include a combination of didactic, small group and clinical experiences. This will determine the types of faculty required to teach and opportunities to choose from a variety of modes of delivery.

기술의 활용 Use of technology

임상실습이 아닌 거의 모든 교육과정이 온라인으로 제공될 수 있다.

Communications technology has developed rapidly to the point where virtually all of the non-practical curriculum could be delivered from the main base. 

그러나 모든 사람이 이러한 방식을 즐기는 것이 아니므로, 혼합할 필요가 있다.

However, in reality not everybody enjoys participation purely through communications technology. Hence there will almost certainly have to be a combination of distant and local delivery. This raises an important philosophical issue: the aim in using communications technology must be to facilitate or enable optimal curriculum delivery. This is a more learner-focused than technology-focused approach, but there are several challenges. 

• One is how to closely align the curriculum with the technology, as there are no clear guidelines and perhaps as many variations as there are curricula. 
• Another is the issue of compatibility of the technology backbones of the partners, as this is unlikely to be complete. 
• Finally, the continuing rapid development of communications technology may mean that regular improvement may be constrained by funding availability.

파일럿 프로그램 Local piloting of the program

As with any new initiative, it is unwise to commence full operation without some piloting and probably a staged introduction. Technology should be trialed well before it can be relied upon. Local curriculum implementation should be piloted, perhaps with a small group of volunteer students in the previous academic year.

프로그램 평가 Program evaluation

시기별 목표가 다를 수 있음

Program evaluation is essential to the success of the program and requires a solid funding envelope. Although the 

• early focus is on evaluation to improve quality, to document student achievement and satisfaction, comparability, and matching rates into postgraduate training, a 
• longer term focus is on measurement of the degree to which the program is meeting its stated mission and goals, as well as identification of unintended outcomes. 

관계자들에게 보고서 제출해야 함

It is imperative to provide regular reports to stakeholders, including future students, on program indicators. 

메인캠퍼스와 지역캠퍼스는 관점이 다를 수 있다.

Main and regional campuses have different points of view about program evaluation. 

• Central university may be focused on their usual parameters of success such as academic achievement of entering students. 
• However, the regional campus may be more interested in the impact on workforce recruitment, and on the community overall.

Planning program evaluation during the hectic start-up days can be challenging. The recruitment of an individual whose position has this as a priority has led to success at many regional sites. Impact on recruitment and workforce appears to take place well before graduation of a cohort – the regional campus itself can act as a magnet for clinicians. Capturing these changes in the first years of the program is important, as the baseline precedes any announcement of the regional campus.

장기적 ROI측정을 위한 전략