자기평가와 자기모니터링의 차이 분석 (Adv in Health Sci Educ, 2011)

Exploring the divergence between self-assessment and self-monitoring

Kevin W. Eva • Glenn Regehr

전문직의 자기조절에 관한 많은 모델이 자신의 기술의 부족함을 알고, 그것을 CPD 활동을 통해서 채워나갈 책임을 요구한다. 그러나 이러한 기대와 달리 자기평가에 대한 많은 문헌들은 자기조절적 전문직이 이 과저을 효과적으로 할 수 있는가에 대한 의구심을 표한다. 이에 대해서 저자들은 자기조절에 관한 문헌에서 표현되는 자기평가에 대한 구조가 문제가 있음을 지적한 바 있다. 본 논문에서 2 개의 연구를 통해서 자기평가와 자기조절의 관계를 살펴보고자 한다.

• 자기평가: a global judgment of one’s ability in a particular domain
• 자기조절: a moment-by-moment awareness of the likelihood that one maintains the skill/knowledge to act in a particular situation

Many models of professional self-regulation call upon individual practitioners to take responsibility both for identifying the limits of their own skills and for redressing their identified limits through continuing professional development activities. Despite these expectations, a considerable literature in the domain of self-assessment has questioned the ability of the self-regulating professional to enact this process effectively. In response, authors have recently suggested that the construction of self-assessment as represented in the self-regulation literature is, itself, problematic. In this paper we report a pair of studies that examine the relationship between self-assessment (a global judgment of one’s ability in a particular domain) and self-monitoring (a moment-by-moment awareness of the likelihood that one maintains the skill/knowledge to act in a particular situation). 

본 연구들은 수행능력과 자기평가와의 상관관계는 낮지만, 자기모니터링과 관련한 척도와는 상관관계가 높음을 보여준다.

These studies reveal that, despite poor correlations between performance and self-assessments (consistent with what is typically seen in the self-assessment literature), participant performance was strongly related to several measures of self-monitoring including: 

• 문제에 대해서 답을 할 것인가 미룰 것인가. the decision to answer or defer responding to a question, 
• 그 결정(답을 할 것인가 미룰 것인가)을 내리는데 걸린 시간 the amount of time required to make that decision to answer or defer, and 
• 정답이 제시되었을 때의 답에 대한 자신감 the confidence expressed in an answer when provided. 

이러한 차이는 자기모니터링과 자기평가에 대한 인지적 기전을 이해하는데 도움이 될 것이며, 교육과 학습에 대한 노력을 어떻게 더 잘 제시해줄 수 있을 것인가에 대한 이해에 도움이 될 것이다.

This apparent divergence between poor overall self-assessment and effective self-monitoring is considered in terms of how the findings might inform our understanding of the cognitive mechanisms yielding both self-monitoring judgments and self-assessments and how that understanding might be used to better direct education and learning efforts.

개개인이 효과적인 자기평가를 할 수 있는가에 대한 의문이 제기되고 있다.

there is now a well established literature that raises doubts about the capacity of individuals to effectively self-assess either personal (Dunning et al. 2004) or professional (Gordon 1991; Boud 1995; Davis et al. 2006) areas of relative strength and weakness. Increasingly it is being recognized that self-assessment as ‘‘a process of personal reflection based on an unguided review of practice and experience for the purposes of making judgments regarding one’s own current level of knowledge, skills, and understanding as a prequel to self-directed learning activities that will improve overall performance and thereby maintain competence’’ (Eva and Regehr 2007, p. 81) is inherently flawed."

Adv Health Sci Educ Theory Pract. 2011 Aug;16(3):311-29. doi: 10.1007/s10459-010-9263-2. Epub 2010 Nov 30.

Exploring the divergence between self-assessment and self-monitoring.

PBL의 세계화에 대해 다시 생각하다: 어떻게 문화가 자기주도학습에 영햐을 주는가 (Med Educ, 2012)

Rethinking the globalisation of problem-based learning: how culture challenges self-directed learning

Janneke M Frambach,1 Erik W Driessen,1 Li-Chong Chan2 & Cees P M van der Vleuten1

교육법은 문화와 사상을 반영한다. 끊임없이 교육법이 전 세계적으로 퍼져나가고 공유되는 현 시점에서 이러한 인식에 대한 여러 문화간 함의를 살펴보는 것이 중요하다.

It is generally acknowledged that education methods reflect cultural and ideological values.1–3 Addressing the cross-cultural implications of this notion is increasingly urgent in view of the continuing dis- semination of education methods around the globe.

문화간 공유하는 가치가 존재한다는 가정이 틀릴지도 모른다는 것에 대한 반대주장은 상당히 무시되어왔다. 세계화는 교육법의 표준화를 가져왔고, 문화적 차이에 대한 고려 없이 여러 문화권에서 활용되고 있다. 의학교육 외 분야에서의 연구를 통해 학생의 학습과 교육적 접근에 대한 선호에 문화간 차이가 있다는 것이 밝혀졌다. 그 결과 '국제적' 교육법에 대한 문화적 근원이 존재할 것이라는 가정은 다른 문화권에서는 적절하지 않을 수도 있다.

Counterarguments that the assumption of shared values across cultures may be false seem to be largely ignored.8 Driven by ideological or other motives, the globalisation movement promotes the standardisation of education methods and practices across cultures, apparently with little regard for cultural differences.9,10 Research outside medical education has revealed differences between cultures in students’ learning and preferences for educational approaches.11–13 Consequently, the cultural originof a supposedly ‘international’ educational approach may compromise its suitability for other cultural contexts.3

서양 문화에 기반을 둔, 학생 중심의 문제 중심의 방법은 진정으로 국제적이지 않을 수 있고, 여러 비-서구권 문화에서 적합성에 문제가 제기된 바 있다. Gwee와 Khoo는 아시아 문화권에서의 태도는 PBL의 교육 원칙과 잘 맞지 않을 수 있으나, 이 차이를 좁힐 수 있는 태도에 대해서도 언급하기도 했다. PBL이 여러 문화권에 걸쳐서 적용가능한가에 대한 실제 연구는 학생과 교수 등을 통한 연구에서 긍정적으로 보여진 적도 있지만, 서양에서 진행되는 PBL과의 차이 혹은 거기서 발생하는 문제도 지적된 적 있다. 대부분의 이러한 연구는 PBL의 도입시기 혹은 도입 직후에 국한되었거나, 단일기관, 단일지역에서 수행되었다.

Rooted in Western culture, student-centred, problem- based methods may not be of a truly international nature3,14 and their compatibility with non-Western cultures has been questioned.15 Gwee5 and Khoo16 pointed to Asian cultural attitudes that might be difficult to reconcile with the educational principles of PBL, but also noted attitudes that might mitigate this discrepancy. The few empirical studies into the cross-cultural applicability of PBL reported positive views among students and staff,6,7,17,18 but also noted problems and assumed differences with Western practice.17–19 Most of these studies were limited to the implementation phase of PBL or shortly thereafter and to single institutions, countries or regions, mainly in Asia.

본 연구에서는 어떻게 문화적 요인이 SDL이라는 PBL의 주 교육원칙에 영향을 주는가에 대해 알아보고자 한다. 이러한 원칙은 '민주주의, 개인주의, 평등주의라는 서양의 사상'에 크게 기반하고 있다.

The present study investigates whether and how cultural factors affect one of PBL’s main educational principles: self-directed learning (SDL).20 It has been argued that this principle relies strongly on ‘Western ideals of democracy, individualism and egalitarian- ism’.21 It is defined here as: 

학생이 교수에 의해 정의된 것이 아니라 스스로 학습활동을 정의하고 참여하는 것에 대한 준비도. '준비도'라는 것은 동기부여 뿐 아니라 적절한 행동기술을 포함한다. 따라서 SDL을 하는 학습자는 지식 습득에 대한 내적 요구가 있으며, 이 요구는 선생님에 의해서 정해진 것이 아니다. 추가적으로 이 학생은 적절한 정보탐색기술을 익혀야 하며, 이는 스스로의 요구를 충족시키기 위한 정보원이 어딘지를 알고 찾는 것이다.

‘…the preparedness of a student to engage in learning activities defined by himself rather than by a teacher. ‘‘Preparedness’’ must be understood as having both a motivational aspect and involving skilled behaviour. Thus, an accomplished self-directed learner experi- ences an intrinsic need to acquire knowledge, not dominated by requirements set by his teachers. In addition, he has mastered the appropriate information seeking skills, that is: he knows where and how to find information resources that would fulfil his need.’22

Culture is defined as the shared motives, values, beliefs and identities of members of collectives.13 

사회문화이론가들은 인간은 환경의 규범과 특징을 내면화하면서 지속적으로 환경에 의해 형성되어가며, 반대로 스스로 가진 생각과 가치를 환경에 외면화하면서 그 환경에 영향을 주고 바꾼다.

Socio-cultural theorists state that humans are continuously influenced and shaped by their environment as they ‘internalise’ its norms and characteristics.26,27 Conversely, humans influence and transform their environment by ‘externalising’ their inner ideas and values.27

자료수집

The semi- structured interviews lasted 1 hour on average and were audio-recorded and transcribed verbatim. Oral and written informed consent was obtained. The participants received a symbolic gift. Purposive sampling ensured the inclusion of male and female students, students from different PBL groups and from the first and third years of training. 

The researchers were briefly introduced at the start of the tutorials and did not participate in sessions. Documents about the implementation and application of PBL were obtained from the key persons. The researchers kept journals in which they recorded additional contextual information. They also reported personal perspectives to create awareness of potential researcher bias. To enhance the trustworthiness of the data, a member check was conducted by asking a sample of the participants to indicate agreement with and comment on a report of preliminary results. The comments were integrated with the data.

Using the thematic approach of template analysis,28 a succession of coding templates, consisting of hierar- chically structured themes, were applied to the data (Fig. 1).

28 King N. Using templates in the thematic analysis of text. In: Cassel C, Symon G, eds. Essential Guide to Qualitative Methods in Organizational Research. London: Sage Publications 2004;256–70.

중동국가에서의 불확실성과 전통

Uncertainty and tradition in the Middle East 

Middle Eastern students expressed more feelings of uncertainty as a cultural factor compared with Dutch and Hong Kong students. Their uncertainty and difficulties in adapting to SDL were related to sharp contrasts between PBL and their prior educational experiences. Rather than feeling motivated, many students felt lost and unable to find appropriate information to address their learning objectives. Uncertainty was related to experiences of traditional, teacher-centred secondary education, but also to a culturally determined focus on tradition. Middle Eastern respondents referred to their society’s respect for the ‘old ways’ and wariness regarding innovations. As they became used to PBL, however, their attitudes changed significantly. Students came to support the principle of SDL and information seeking became less problematic, although students still felt PBL was not easy and wanted more guidance:

Having experienced information searching and self- study in secondary school, Dutch students had less difficulty in adapting to SDL. In addition, Dutch culture places less value on tradition. Although the Dutch students were less uncertain, they required time to develop information-seeking skills and they generally preferred tutors who provided clear guidance. A particular problem for them concerned determination of the depth and breadth of the knowledge to be attained. Dutch, Hong Kong and, particularly, Middle Eastern students tried to cope with uncertainty and independence by asking senior students for advice and materials. Although it reduced insecurity, this strategy discouraged them from depending upon themselves for their learning:

혼합과 위계의 홍콩

Hybridism and hierarchy in Hong Kong 

From the outset, finding information was less difficult for Hong Kong students. As topics of tutorials were also covered in lectures in the hybrid curriculum, identifying learning needs and developing information-seeking skills were less relevant to Hong Kong students. They showed little awareness that PBL was intended to foster SDL. Whereas the lectures covered the basic sciences, the Hong Kong tutorials focused more on clinical reasoning skills. By contrast, the Dutch and Middle Eastern students had to rely on tutorials for most of their knowledge. The Hong Kong students often felt the tutorials repeated the content of lectures, which some appreciated as providing a useful opportunity for revision and a chance to apply their knowledge to a clinical case, but others considered a waste of valuable study time:

Some Hong Kong students were anxious about multiple interpretations that might come up during tutorial discussions because these created uncertainty about the ‘truth’ and they were hesitant about trusting their peers’ statements. This reflected their experience of a teacher-centred secondary education, as well as the culture of a hierarchical society in which knowledge and authoritative statements about the ‘truth’ are expected to come from professors or experts who represent persons of higher status. Students were not used to having to rely entirely on themselves for their learning. They also attached greater value to tutorials that were facilitated by expert clinicians rather than by non-experts. Although Dutch students also preferred facilitation by expert tutors, they were comfortable relying on their peers. The impact of hierarchy was also evident in the Middle Eastern school and manifested in students’ experiencing of anxiety about the requirement to search independently for the ‘truth’. However, by Year 3, student anxiety in the Middle East school had abated, whereas student anxiety in Hong Kong showed little difference between the years:

성취와 평가에 대한 문화간 차이

Achievement and assessment across cultures 

Middle Eastern and Hong Kong students characterised themselves and their respective societies as competitive and described themselves as striving for success and to be the best. They felt pressured to pass examinations and rank among the top students:

Dutch students were also examination-focused, although their responses during interviews suggested a lower level of culture-related focus on achievement and success compared with the other two cohorts. The general feeling among the three groups of students was that they valued PBL only for its contribution to their examination preparation. This depended on examination content. In Hong Kong, examination content was mainly determined by lectures. In the Middle Eastern and Dutch schools, it depended more on PBL tutorials. However, particularly in the Middle Eastern school, the inclusion of additional topics caused students to concentrate on these predetermined additional topics and their lecture notes more than on identifying and addressing their individual learning needs. Even if they supported and understood the principle of SDL, achievement and assessment took priority, directing their attention and efforts away from SDL to exam- ination content:

Discussion

보통 불확실성, 전통, 위계, 성취에 대한 강조는 비-서구권에서 서구권보다 더 두드러지는 특징이다. PBL과 비-서구 문화권 사이에서 발생하는 불일치는 여기서 발생하는 것일 수 있으며, PBL을 곧바로 이들 문화권에 적용하는 것을 어렵게 한다.

Uncertainty, tradition, hierarchy and achievement have often been identified as more prominent in non-Western than in Western cultures.29–31 This suggests a certain incongruity between PBL and non-Western cultures, which complicates the straightforward transfer of PBL to such cultural contexts.

그러나 문화적 요인이 모든 것을 설명해주지는 않았고 다른 것도 있다.

However, cultural factors clearly do not explain all of the discrepancies in findings between the respective contexts. Several contextual factors, such as a 

• traditional, teacher-centred secondary education, 
• a hybrid curriculum and 
• examination content not covered during PBL sessions 

further complicated students’ development of SDL skills. 

예를 들면...홍콩 고등학교 교육은 SDL기회가 거의 없음. 학생들은 선생님-의존적이 됨.

For example, the secondary school education system in Hong Kong is very much based on knowledge acquisition and rote learning to pass examinations. Because teachers and recommended textbooks serve as the main sources of information, there is little opportunity for SDL. Therefore, it is not surprising that current Hong Kong medical students remain dependent on teachers and lectures for their learning. However, this may change in the future in response to education reform taking place in Hong Kong high schools, which emphasises SDL by students as a major educational goal.

SDL과 PBL기술이 그것이 적용되는 context에 크게 영향을 받는다는 이전 연구를 뒷받침한다. PBL이 도입되었다고 SDL이 자연적으로 발생하지는 않는다. 촉진적 환경을 만들어주기 위해서는 정교하게 계획되고 집중적 노력이 필요하다. 실제로 1학년 학생을 적절한 가이드 없이 PBL의 독립적 학습환경에 넣어두면 살아남기 위해서 오히려 튜터, 사전에 정해진 학습목표, 단순 암기에 지나치게 의존하게 된다. 우리 연구에서도 세 문화권 학생은 서로 정도의 차이는 없었지만 비슷한 행동패턴을 보였는데, 불확실성을 줄이려고 한다거나, 선배에게 상담을 한다거나, 튜터에게 가이드를 요청한다거나, 시험 내용에 초점을 맞추는 것 등이 공통적이었다.

Our findings support earlier comments that the development of SDL and other PBL skills depends heavily on the context in which PBL is applied.21 Research suggests that SDL does not occur automatically when PBL is implemented. Carefully considered and focused efforts are needed to shape a propitious context.32,33 In fact, exposing Year 1 students to the independent learning environment of PBL without providing them with adequate guidance may, rather than promoting the development of SDL skills, cause them to become severely dependent on tutors, predetermined learning objectives and on rote learning in order to ‘survive’.32,33 This is supported by our findings that students across three different cultures, albeit to different degrees, mentioned sim- ilar behaviours, needs and preferences with regard to alleviating uncertainty, consulting senior students, asking for tutor guidance and focusing on examination content.

가능한 대안 중 하나는 SDL을 PBL을 위한 도구로 활용하고, PBL의 목표로 인식하게 하는 것이다. 점진적으로 SDL에 노출시켜 주는 것이, 즉 1학년에는 가이드와 지원 체계를 강력하게 하는 것이, 궁극적으로 SDL 기술을 더 발전시킬 것이다. 특히 중고등학교 교육이 교사 중심이고 hybrid 접근법이 없는 경우에, 그리고 불확실성, 전통, 위계, 성취가 강조되는 문화권에서 중요하다.

A possible solution might be to strike a balance between using SDL as a means to PBL and perceiving it as an end of PBL.33 Gradual exposure to SDL, with relatively strong guidance and support in the first year, might ultimately yield the development of more SDL skills.33 Our findings suggest that this is particularly relevant in contexts in which secondary education is teacher-centred, no hybrid approach is followed, and the cultural factors of uncertainty, tradition, hierarchy and achievement are valued highly.

그럼에도 불구하고 세 문화권 학생들 모두 1학년에서 3학년으로 올라감에 따라 SDL의 원칙을 내면화하는 것으로 보였다. 

Despite the challenges, however, students across the three cultures increasingly internalised the principle of SDL as they moved from Year 1 to Year 3. 

• The Middle Eastern students made substantial progress from initial uncertainty to preparedness to determine their own learning activities and find relevant infor- mation. 
• Because of their pre-university experiences in SDL, the progress of the Dutch students was less marked, but still noticeable. 
• Hong Kong students seemed to quickly adapt to the PBL learning environment and to develop clinical reasoning skills, but were less stimulated to develop SDL skills in terms of determining learning objectives and con- sulting different information sources. 

이러한 결과는 학생들이 PBL 과정에 익숙해짐에 따라서 SDL 기술이 발전한다는 기존의 연구와도 일관된 것이다. 따라서 PBL이 비록 여러 문화권에서 곧바로 적용하기는 어려울지라도, 적용될 수 없다는 결론을 내리는 것 또한 옳지 않다.

These findings would appear to be consistent with those of studies reporting that SDL skills develop naturally as stu- dents become used to the PBL process and curricu- lum.33,34 Thus, although PBL may not be cross- culturally applicable in a straightforward way, it would be wrong to conclude that it cannot be applied across cultural contexts as practice continues to prove.

근본적인 질문은 PBL이 애초에 국제적으로 적용되어야 했었냐는 것이다. 의학교육이 전 세계적으로 개혁이 필요한 것은 사실이지만, 한 해법이 모든 곳에 적용되어야 하는냐는 논란의 여지가 있다. 현재는 서구에서 유래한 학생중심 교육법이 '국제적' 기준으로 대표되고 있다. 그러나 이 연구에서 이러한 방법들의 문화권간 적용 가능성에 의문을 가질 수 있음을 보여준다. 학생 중심의 문제 중심의 방법을 적용하면서 문화적 도전에 맞닥뜨리느니, 오히려 그 맥락에 가장 잘 맞는 대안을 찾거나 만드는 것이 나을 수도 있다. 현재의 '국제 기준'에 대한 움직임을 감안하면 이는 쉽지 않다. 그러나 아시아 지역의 영향력이 높아지고, 다른 지역도 발달함에 따라 미래의 의학교육의 지형도 바뀔 것이다.

A more fundamental question is whether PBL should be globalised in the first place. It is true that medical education worldwide is in need of reform, but whether one solution should be applied to all contexts is debatable. Currently, student-centred methods originating in Western culture seem to represent an ‘international’ standard. Yet, as this study confirms, the cross-cultural applicability of these methods may be questionable. Rather than taking on the cultural and contextual challenge of adopting student-centred, problem-based methods, it might be wiser for medical educationalists to rise to the challenge of exploring or creating alternatives that best fit their particular context. Given the current movement towards the development of ‘international standards’, this is a major challenge indeed. However, the rising influence of the Asian region and rapid developments in other parts of the world may imply changes in the future landscape of medical education.

11 Tweed RG, Lehman DR. Learning considered within a cultural context: Confucian and Socratic approaches. Am Psychol 2002;57 (2):89–99. 

12 Li J. Mind or virtue: Western and Chinese beliefs about learning. Curr Dir Psychol Sci 2005;14 (4):190–4. 

13 Joy S, Kolb DA. Are there cultural differences in learning style? Int J Intercult Relat 2009;33 (1):69–85.

8 Hodges BD, Segouin C. Medical education: it’s time for a transatlantic dialogue. Med Educ 2008;42 (1):2–3. 

9 Karle H, Christensen L, Gordon D, Nystrup J. Neo- colonialism versus sound globalisation policy in medi- cal education. Med Educ 2008;42 (10):956–8. 

10 Hodges BD, Maniate JM, Martimianakis MA, Alsuwai- dan M, Segouin C. Cracks and crevices: globalisation discourse and medical education. Med Teach 2009;31 (10):910–7.

31 Al Kadri HM, Al-Moamary MS, Magzoub ME, Roberts C, van der Vleuten CPM. Students’ perceptions of the impact of assessment on approaches to learning: a comparison between two medical schools with similar curricula. Int J Med Educ 2011;2:22–52.

SDLRS: 요인분석 (Med Educ, 2005)

The Self-Directed Learning Readiness Scale: a factor analysis study

J Dennis Hoban, Sonya R Lawson, Paul E Mazmanian, Al M Best & Hugo R Seibel

의학에서 SDL은 중요하다. ABMS, RCPSC, WFME 등은 평생학습을 수련기간동안 반드시 평가해야 하는 전문성 중 하나로 보았다.

Self-directed learning is important in medicine where knowledge is continuously changing and a wide- range of patient problems is a constant.1 The American Board of Medical Specialties, The Royal College of Physicians and Surgeons of Canada, and the World Federation for Medical Education describe life-long and self-directed learning as professional characteristics that should be evaluated in the train- ing of physicians.2–4

1970년대 후반, Guglielmino가 SDL에 대한 준비도를 측정하는 도구를 개발하였다. Delphi 연구를 통해서 타당도를 마련했으며, 이러한 귀납적 접근법을 통해서 SDL에 대한 준비도를 개념화하고 다음과 같은 정의를 내렸다.

In the late 1970s, Guglielmino developed an instru- ment to measure self-directed learning readiness.5 A Delphi study was conducted to gain expert consensus on the characteristics of the self-directed learner. In the case of the SDLRS a foundational effort to ensure content validity involved basing the items on a Delphi panel s perception of the charac- teristics of an individual with a high level of readiness for self direction in learning’6 (p. 213). Using this inductive approach to conceptualise self-directed learning readiness, she offered a definition of the self-directed learner: 

• one who exhibits initiative, independence, and persistence in learning; 
• one who accepts responsi- bility for his or her own learning and views problems as challenges, obstacles; 
• one who is capable of self-discipline and has a high degree of curiosity; 
• one who has a strong desire to learn or change and is self-confident; 
• one who is able to use basic study skills, organize his or her time and set an appropriate pace for learning, and to develop a plan for completing work; 
• one who enjoys learning and has a tendency to be goal-oriented.5 (p. 73). 

Guglielmino는 이러한 특질과 관련된 문항을 개발해서 41문항으로 된 원본을 만들었고 PCA w/ varimax rotation으로 8개 요인 구조를 도출했다.

Guglielmino developed items that related to these qualities and produced her original version of the SDLRS consisting of 41 items. She tested it on a group of 307 US high school juniors and seniors, college undergraduates, and adults in continuing education courses. She reported that principal com- ponent analysis (PCA) with varimax rotation yielded an 8-factor structure. She labeled these factors: 

(1) openness to learning opportunities; 

(2) self- concept as an effective learner; 

(3) initiative and independence in learning; 

(4) informed acceptance or responsibility for one’s own learning; 

(5) love of learning; 

(6) creativity; 

(7) future orientation; and 

(8) ability to use basic study skills and problem solving skills.5 Later she modified the SDLRS to include 58 items.6 

SDLRS는 널리 사용되긴 했지만 비판도 많았다. 우선 타당도에 대한 Field의 지적이 있었다. Bonham도 구인타당도에 대한 의문을 삼았다. SDLRS점수가 낮다는 것은 두 가지를 의미하는데, (1) 공부를 싫어하거나 (2) 다른 사람에 의해 지도받는 것을 좋아하거나. 

The SDLRS has been widely used but also criticised. Field questioned the validity of the scale as a measure of readiness for self-directed learning.7 Responding to Field’s criticism, Guglielmino wrote that readiness in the scale title is a measure of an individual s current level of readiness to engage in self-directed learning ‘‘with the implication that this level may change’’’8 (p. 236). Bonham also questioned the construct validity of the SDLRS, suggesting that low scores on the SDLRS could mean two things: a dislike for learning or one’s preference for his or her learning to be directed by another.9 Brockett and Hiemstra stated, …the evidence is rather convincing that early concerns raised about certain items of the scale are warranted 10 (p. 73). 

이후 연구에서 SDLRS가 실제로 측정하는 것에 무엇인가에 대한 결과가 엇갈렸다. 

Subsequent empirical work yielded mixed results regarding what the SDLRS actually measured. 

• Mourad and Torrance administered the 58 itemSDLRS to a random sample of 684 K-12 students enrolled in a programme for gifted children at the University of Georgia.11 Their PCA suggested an 8-factor model; they concluded, more studies are needed to validate the scale using different samples (p. 102). 
• Field’s SDLRS study involved 244 adult students in Sydney, Australia.7 Using common factor analysis he identified 4 factors but then concluded that the scale measures a construct that is homogeneous 7 (p. 138). The single construct was love and enthusiasm for learning. 
• Bligh12 conducted an SDLRS study with medical trainees (n ¼ 216) during their medical education preparation in the UK. His PCA yielded 3 major factors: enthusiasm for learning, positive self- concept as a learner, and orientation to learning. 

Field의 연구와 다르게 여기서 언급한 모든 연구는 PCA를 사용했다. PCA와 Factor analysis는 공통점이 있지만, 다른 결과를 내놓기도 한다. 따라서 Field의 분석을 다른 연구자들의 분석과 비교하기는 어렵다. 실제로 McCune은 Field의 연구에 대해서 'PCA와 CFA의 결과 차이를 알아야 한다'라고 지적했다. 연구자들은 EFA를 위해서 종종 PCA를 활용한다. 그러나 우리는 Field의 연구가 Exploratory 하다고 생각한다. 여기서 제시한 연구들은 SDLRS에 깔린 구조를 알아내고자 하는 것이지만 PCA는 그러한 목적의 분석방법이 아니다.

With the exception of Field’s study, all the investiga-tions reported herein employed PCA. While PCA andfactor analysis methods may bear some similarities, they tend to produce different results. Thus it is challenging to compare Field’s factor structure with the others’ component structures. Indeed McCune criticised Field along these lines when she wrote, 'Also, Field should realise that if Guglielmino used principal component analysis while he used commonfactor analysis, their results should differ 13' (p. 245).Researchers often use PCA when conducting explor-atory factor analysis (EFA). We believe even Field’s study was exploratory in nature though McCune offered a competing view when she wrote, He statesthat ‘‘eight factors are sought’’ which I assume is his attempt to portray his analysis as a confirmatory factor analysis 13 (p. 245). PCA reduces a large set of items into smaller components and accounts for all ofthe variance among the items, but the studies we reported using PCA were trying to identify underlying structures of the SDLRS. PCA is not designed for that purpose. Preacher and MacCallum explain the distinction between PCA and exploratory factor analysis.14 

PCA yields observable composite variables (com- ponents), which account for a mixture of common and unique variance (including random error). The distinction between common and unique sources of variance is not recognised in PCA, and no attempt is made to separate unique variance from the factors being extracted. Thus, components in PCA are conceptually and mathematically quite different from factors in EFA.14 (p. 20). 

West와 Bently는 CFA를 이용해서 SDLRS척도를 분석하였다. orthogonal solution은 부적절하다는 결론을 내렸다. 6개 요인을 밝혔다. 이 중 3번 요인이 reverse scored item으로만 거의 이뤄져있었다. 또한 1st-order factor가 포함될 가능성을 추측해냈다.

West and Bentley examined the underlying SDLRS measurement model using confirmatory factor ana- lysis (CFA).15 Their study was conducted with 439 K-12 Tennessee teachers and administrators. The analysis concluded that an orthogonal solution to the SDLRS measurement model was not adequate. More importantly, they reported that a highly correlated 6-factor model best described the underlying theory. The 6 factors were: (1) love of learning; (2) self- confidence as a learner; (3) openness to challenge; (4) inquisitive nature; (5) self-understanding; and (6) acceptance of responsibility for learning. Inter- estingly, factor 3 contained mostly reverse scored items; yet, the possibility of reverse scored methods variance was not reported. Finally, they conjectured that the first order factors could be subsumed under a single factor characterising a higher order structure.

이들 연구에서 무엇을 배울 수 있는가? 첫째, 거의 모든 탐색연구가 reverse score item으로만 이루어진 요인이 있을 가능성을 제시한다. 그러나 이 것에 대해서 충분히 설명한 연구는 없다. 둘째, 거의 모든 연구자들은 EFA를 활용했다. CFA는 1st order factor들 간의 관계를 설명하는 2nd order factor를 찾게 해준다. 

What did we learn from the literature? First, nearly every exploratory study concluded that there was a component ⁄ factor that consisted of reverse scored items including Guglielmino’s5 original analysis. These items included negative statements about a high self-directed learner or a positive statement about a low self-directed learner. Not 1 author fully explained this phenomenon. Second, all researchers used EFA techniques except West and Bentley who used CFA methods. CFA allows the investigator to specify a second order factor to account for relationships among first order factors. In addition, it provides the researcher with information for disen- tangling random and systematic (method) error variance.16 For example, reserves scored items may be investigated as a source of method variance.

SDLRS의 psychometric nature를 연구하기 위해서 다음과 같이 했다.

To study the psychometric nature of the SDLRS for entering medical students, we collected SDLRS data from 972 students and conducted an EFA study with half the data and confirmed the model it produced with a CFA study using the other half of the data. Preacher and MacCallum emphasised the need for making good decisions in the process of conducting exploratory factor analysis.14 They were particularly concerned about using PCA, retaining components with eigenvalues greater than one, and using varimax rotation; a bundle of procedures affectionately termed Little Jiffy .14 This potentially limited approach was used in most of the previous SDLRS factor analysis studies we reviewed. We followed Preacher and MacCallum’s guidelines14 in our factor analysis.

17개 문항이 reverse scored되었다. 

Seventeen of the items are reverse scored. According to Guglielmino, wordingwas reversed in some of the items to prevent the response set of acquiescence (agreeing with all theitems).5

EFA 방법

Exploratory factor analysis – method 

'만약 요인들의 관계를 모른다면, 서로 완전히 독립이라고 가정할 이유가 없다. 따라서 oblique rotation을 하는 것이 안전하다.

We used Principal Axis Factor Analysis SPSS 11.0 for Windows to extract factors. To decide what factors to retain we used the scree plot, results from previous studies, and our comfort with the extracted factors. We decided to use an oblique rotation (Promax) based upon the West and Bentley study and Preacher and MacCallum’s argument, …if the researcher does not know how the factors are related to each other, there is no reason to assume that they are completely independent. It is almost always safer to assume that there is not perfect independence, and to use oblique rotation instead of orthogonal rotation 14 (p. 26). Individual loadings of 0.30 or greater were used in the factor designation. Extracted factors were examined and named based on an analysis of the items loading on each factor. Cronbach a was used to estimate the internal consistency of the items consti- tuting a factor.

EFA 결과

Exploratory factor analysis – results

5번째 요인이 모두 reverse scored item이었다.

Interestingly, the fifth factor measured all reverse scored items. This phenomenon was also reported by Guglielmino5, Field7, and Mourad and Torrance.11 Guglielmino noted her PCA showed that all items in factor 1 of her original 41 item version were negative statements.5 She speculated that it is possible that the factor also includes an avoidance of agreement with negative statements 5 (p. 61). Field7 and Mourad and Torrance11 also reported a factor in which items were phrased so that they had to be reverse scored. Mourad and Torrance offered 2 explanations for this factor (1) an attitude toward negative state- ments and (2) preference for complex and ambi- tious situations 11 (p. 99). Field labelled the factor containing all negatively worded items as facility with negatively phrased items 7 and argued it was not related to readiness for self-directed learning.

반복적으로 나타난 reverse scored phenomenon

Because the reverse scored phenomenon has been repeatedly reported in the literature we decided it was worth analysing in the confirmatory phase of our study.

We hypothesised a series of three models (Models ¼ A, B, C) based on the logic provided by Anderson and Gerbing.21

Model A is a 4-factor confirmatory model that represents the substantive factors derived from the previous EFA.

Model B is a 5-factor confirmatory model that represents 4 correlated substantive factors and an orthogonal reverse coding method factor that loads on reverse scored items from all substantive factors.

Next, Model C, a confirmatory higher order model was developed and evaluated. This model includes a higher order factor that is presumed to account for the 4 first order factors examined in Model A, and like Model B, includes a reverse scoring factor.

CFA 결과

Confirmatory factor analysis – results

Guglielmino는 귀납적 접근법으로 SDLRS를 만들었다. 그녀는 Delphi를 활용했으며, SDL에 바람직하거나, 필요하거나, 필수적인 것을 고려해서 만들었다고 확실히 언급했다. 또한 상황적, 태도적 문항이 필요해서 Delphi와 문항의 1:1 대응은 어려웠다고 했다. 본 연구에서 SDLRS는 Guglielmino가 말한 상황적, 태도적 특징을 충분히 측정하지 못하고 있는 것으로 보인다.

Guglielmino used an inductive approach to develop the SDLRS. She made clear, Results of the Delphi survey were used as a guideline in the construction of items for the scale. Characteristics which emerged from the survey with a rating of desirable, necessary,or essential were considered for inclusion. 5 (p. 37). She explained A one-to-one correspon- dence between SDLRS items and characteristics selected by the Delphi survey was not possible, since situational and attitudinal items were desired. 5 (p. 38). In our study, the SDLRS instru- ment did not fully measure these characteristics orsituational and attitudinal constructs that Gugliel- mino specified. 

본 연구의 EFA에서 4개의 요인을 밝혔다. McCune은 요인분석이 샘플에 따라서 다른 결과를 보일 수 있다고 했다. 따라서 (특히 다른 연구에서 PCA를 활용했다는 점에서) 다른 집단이나 다른 의대생의 결과와 다른 것에 놀라지는 않았다.

Our EFA study produced 4 acceptable factors. SinceMcCune cautioned that factor analysis studies of thesame instrument could yield different results depending on the samples used,13 we were not surprised that our EFA factors varied somewhat from the structures reported for other medical students12 and other populations,5,7,11,15 especially since many of our reviewed studies used PCA.

여러 연구들이 reverse scored item으로만 구성된 요인이 있음을 밝힌 바 있는데, 그 이상에 대한 연구는 없었다. SDLRS는 17개의 그러한 문항이 있다. 점점 더 많은 연구들이 reverse coded item이 내적일관성신뢰도를 떨어뜨리며 factor structure의 해석을 어렵게 함을 보여준다.

While several studies5,7,11 noted the presence of a factor comprising all or mostly reverse scored items, we learned that the reverse coding method variance had not been considered or further explored. The SDLRS contains 17 items that are reverse scored. A growing body of evidence concludes that reverse- coded items may weaken the internal consistency reliability of test score22–24 and impair interpretation of the factor structure.25–27

Marsh는 부정형으로 서술된 문항의 효과를 없애는 가장 쉬운 방법으로 긍정형 문장만 사용하는 것을 제시했다. 이것이 측정 전문가의 권고사항에 반하는 것이긴 하나, reverse coded item과 관련된 문제에 따른 제약이 있는 것은 분명하다.

Marsh suggests that the easiest way to eliminate the effects of negatively worded items is to use only positively worded items… 29 (p. 817). Although this approach runs counter to the recommendations of measurement experts, the problem of method vari- ance associated with reverse coded items suggests a limitation of including reverse scored items in the SDLRS.

여기서 얻을 수 있는 결론은 Guglielmino가 척도를 개발하기 위해 노력한 것은 맞지만 그 SDL척도는 부족해보인다. 우리 연구에 따르면 SDLRS는 다음의 네 가지를 측정한다. 

What can we conclude from our study of the SDLRS? We acknowledge Guglielmino’s efforts to develop a practical instrument for measuring self-directed learning readiness. Her carefully constructed approach to generating the instrument appears appropriate; yet, the SDLRS apparently falls short of measuring characteristics that Guglielmino deter- mined were associated with self-directed learning. With our two samples, the SDLRS measured the respondents’ perceptions of how often they felt positively about: 

• (1) learning being a tool for life; 
• (2) their self-confidence in their abilities and skills for learning; 
• (3) taking responsibility for their own learning; and 
• (4) their curiosity. 

SDL을 잘 하는 학생이 이러한 특징을 많이 가진 것은 직관적으로 말이 되는 것으로 보이지만, 이러한 인식이 SDL행동을 유도한다고 믿을 근거 또한 부족하다. Guglielmino는 원래 Delphi를 통해서 SDL학습자의 특징을 찾아냈을 뿐이고, 이에 대한 어떤 이론적 기반도 제시하진 않았다.

While it makes intuitive sense that self-directed learners would perceive themselves as holding these characteristics in abundance, there is no reason to believe that these perceptions would predict self-directed learning behaviour. Guglielmino identified only characteris- tics of self-directed learners from her original Delphistudy; she offered no theory of self-directed learningor of readiness.

지난 25년간 SDL에 대한 연구와 컨퍼런스가 많았지만 SDLRS가 바뀌지는 않았다. 의학에서 SDL에 대해서 SDL에 영향을 주는 조건에 대한 것도 설명해야 한다. 신경생물학의 최근 연구를 보면 다음의 것들을 중요시하고 있다. 학생과 의사들은 단순히 학습에 대해서 새로운 기술을 습득하는 것으로 충분하지 않다. 많은 경우 자신과 자신의 일과 지속적 전문성 개발에 대한 생각을 다시 할 것을 요구받는다. 우리는 Baveye가 말한 것처럼 SDL를 단순히 SDL에 대한 인식을 측정하는 것에서 관찰가능한 SDL노력이 어느정도인가를 측정하는 것으로 바뀌어야 한다고 생각한다.

Even as conferences and books exploring self-directed learning proliferated during the past 25 years, there appears to be no change in the SDLRS. In considering self-directed learning in medicine, one must account for the current research as well as conditions that influence the performance of self-directed learning. Newer studies extend fromthe neurobiology of aging and the role of cognition in making practice chan- ges30 to the social psychology of the physician’s changing work environment, the importance of phy- sicians’ peers, and the accountability schemes and financial incentives built into medical practice.31 Students and practitioners are not being asked merely to take on new skills or to adjust their attitudes toward learning. Many are being asked to rethink the way they see themselves, their work, and their ongoing professional development. We agree with Baveye32,33 who suggests that the study of self-directed learning should be reoriented to an entirely new direction, away from simple measures of perceptions of self- directed learning to observed self-directed learning endeavours, apropos of the 21st century.

The Self-Directed Learning Readiness Scale: a factor analysis study.

자기조절학습과 의과대학 학업성적의 관계(Med Teach, 2015)

Self-regulated learning and academic performance in medical education

SUSANNA M. LUCIEER1, LAURA JONKER2,3, CHRIS VISSCHER2, REMY M. J. P. RIKERS4,5 & AXEL P. N. THEMMEN1,6

의료전문직은 변화하는 사회에 맞춰서 높은 수준을 유지해야 한다. CME에서 배우기 위해서는 아래와 같은 것을 해야 하며, 요약하면 의사들은 자기조절적 학습자가 되어야 한다는 것이다. 이것은 학습과정에 있어 행동, 메타인지, 학습동기 등이 주도적이어야 함을 뜻한다.

The medical profession has to ensure that high standards in providing patient care are repeatedly being met in the context of a rapidly and constantly changing medical world (Brydges &Butler 2012; Bjork et al. 2013). This means that medical doctors have to stay updated with the developments in their field of expertise and have to maintain their competencies (Greveson& Spencer 2005; Artino et al. 2012; Brydges & Butler 2012;Premkumar et al. 2013). To be able to benefit and choose from the many opportunities of continuous medical education,medical doctors have to define their own learning needs, set personal goals and engage in the most appropriate learning activities (Lycke et al. 2006; Brydges et al. 2012; Premkumaret al. 2013). In short, medical doctors have to be self-regulated learners, which means that they have to be behaviorally, meta-cognitively and motivationally proactive in their learning process (Zimmerman 1986; Wolters 1998; Jonker et al. 2010).

Ertmer와 Newby에 따르면, 자기조절적 학습자는 다음의 것을 할 수 있다. 다른 연구자는 자기조절학습의 동기와 관련된 요인을 언급했는데 아래와 같다. 그러나 이러한 요인들은 학습자가 스스로 사용할 동기가 없으면 별로 가치가 없고, 그래서 두 가지를 추가했다. (노력과 자기효능감)

According to Ertmer and Newby (1996), self-regulated learners are individuals who are able to 

• plan their study behavior, 
• monitor their progress, 
• reflect upon, and 
• evaluate the entire learning process. 

Other researchers also highlighted the importance of motivational components in self-regulated learning (Hong & O’Neil 2001; Sitzmann & Ely 2011). They argued that one may be 

• able to plan, 
• monitor, 
• reflect upon, and 
• evaluate his or her learning behavior, 

but that these competencies are of little value when one is not motivated to employ them. Therefore, they added two subcomponents of motivation to the concept of self-regulated learning, i.e., effort and self-efficacy. 

• Effort is crucial to reach the goals self- regulated learners have set, and 
• self-efficacy is important since one needs to have trust in his or her own potential in order to complete a task (Hong & O’Neil 2001; Sitzmann & Ely 2011).

불행하게도, 자기조절적 학습 기술은 의과대학기간에 늘 강조되는 것은 아니다. 어떤 연구에서는 의과대학동안 학생들의 자기조절학습능력이 향상된다고 나오나, 일부 졸업생들은 그렇게 잘 준비되지 않았다고 느낀다. 

Unfortunately, self-regulated learning skills are not always emphasized during medical school (Artino et al. 2012). While studies showed that students do develop self-regulated learning skills during medical school (Loyens et al. 2008), some graduates feel uncertain and unprepared to do so (Artino et al. 2012). Therefore, it is important to investigate to what extent medical students’ self-regulated learning skills change during their education.

자기조절학습은 학업능력을 가장 잘 예측해주는 요인으로 나타나기도 한다. 자기조절학습은 능동적인 학습과정으로서 목표를 설정하고 효과적인 학습전략을 개발하는 것이다. 이 단계는 학습능력의 정신능력을 변화시키는데, 예컨대 목표설정, 학습전략 개발, 향상과 효과성의 모니터링 등이다. 어떻게 학습발달과정을 모니터링하고, 학습 행동을 조절하고 적응시킬 것인지가 진정으로 효과적인 학습자의 요건으로 여겨진다. 비록 지난 연구들이 자기조절 학습이 반드시 높은 성취를 위해 필요한 기술은 아니라고 하기도 하나, 자기조절적 학습자가 더 효과적인 학습자라는 것이 보여진 바도 있다.

It has also been shown that self-regulated learning is one of the best predictors of academic performance (Pintrich & Degroot 1990). Self-regulated learning is viewed as a proactive learning process that is used to set learning goals and develop effective strategies for learning (Zimmerman 2008). This process helps people to transform mental abilities in academic skills, such as setting goals, developing learning strategies, and monitoring the progress and effectiveness of their learning (Zimmerman 2002, 2008). Knowing how to monitor the progress of your learning and how to control and adapt your learning behavior, is seen as a requirement for being a truly effective learner (Ertmer & Newby 1996; Bjork et al. 2013). Although research suggests that it is not necessary to use self- regulated learning skills for high achievement (Ablard & Lipschultz 1998), it has been shown that self-regulated learners are more effective learners (Nota et al. 2004; Toering et al. 2009) who get more out of their potential (Zimmerman 1986) and attain higher grades during high school (Nota et al. 2004) and in college (Ablard & Lipschultz 1998).

In this study, the Self-Regulation of Learning Self-Report Scale (SRL-SRS) is used. This questionnaire contains six subscales: 

• planning, 
• monitoring, 
• evaluation, 
• reflection, 
• effort, and 
• self-efficacy, 

following the theories of Ertmer and Newby (1996) and Hong and O’Neill (2001).

Setting

Participants

Instruments

The Self-Regulation of Learning Self-Report Scale (SRL-SRS) was used to investigate the students’ level of self-regulated learning. The SRL-SRS contains 50 items on a 4- or 5-point Likert scale, depending on the subsection of the questionnaire. Following the theory described by Ertmer and Newby (1996) and Hong and O’Neill (2001), the questionnaire comprises six subscales of original English-language questionnaires: planning, monitoring, evaluation, reflection, effort, and self-efficacy. An example of a question in the subscale monitoring is: ‘‘While making an assignment, I check my progress,’’ and an example from the subscale effort is: ‘‘I keep trying to finish my assignment, even when I find the assignment extremely difficult’’. The questionnaire has been compiled and validated in a Dutch study (Toering et al. 2012). The questionnaire was originally created for high school students. Therefore, in this study, minor changes were made in a few questions, e.g., the term homework was replaced by study assignments.

Measurements of academic performance

Data analysis 

Data were analyzed with the use of IBM SPSS AMOS version 18.0 (SPSS, Inc., Chicago, IL) and IBM SPSS Statistics version 21.0 (SPSS, Inc., Chicago, IL). Confirmatory factor analysis and Cronbach’s alpha were used to investigate whether the constructs to of the questionnaire fitted the model and measure the internal consistency of the factors. A one-way ANOVA was performed to compare the level of self-regulated learning skills of the first and third-year medical students, a p value of 50.05 was considered significant. For the subscale reflection, Welch F was calculated since equal variances could not be assumed. Effect sizes, eta squared, were converted where 0.01, 0.06 and 0.14 indicate a small, medium, and large effect, respectively (Cohen 1988; Lakens 2013). The correlation between the self-regulated learning skills and the measures of academic performance were calculated with Pearson correlations and multinomial logistic regression analysis. Here, given the multiple comparisons, a more conservative p value of 50.01 was considered significant.

결과

Validation of the questionnaire

Change of self-regulated learning skills

Correlation with academic performance

자기조절학습은 GPA에서 작은 부분만 설명한다.

Multinomial logistic regression analyses showed that self- regulated learning skills explained a small proportion of the variance in GPA among first-year medical students: R2 ¼0.086, Model 2 (18) ¼1592.612, p50.001 as well as some of the R2 ¼0.105, Model 2 variance of the third-year students: (18) ¼38.735, p ¼0.003

1학년과 3학년을 비교하면 의과대학기간동안 자기조절 학습 기술은 별로 달라지지 않았다.

Concerning the first question, we hypothesized thatstudents’ self-regulated learning skills would change duringmedical school. However, we found that the levels of mostself-regulated learning skills did not differ between the firstand third year at medical school, except reflection, which washigher in the third year.

이는 어쩌면 최고의 학생들만 의과대학에 입학하기 때문이 ceiling effect 때문일 수도 있다.

It is however possible that, since only the best students are accepted for medical school (Razack et al. 2012) these students already score relatively high at entrance, and therefore show little development of self-regulated learning during medical school itself (i.e. ceiling effect)

In the third year of medical school, only effort was to some extent related to performance differences.

cross-sectional design이라는 한계가 있으나, 그룹간 age, gender가 비슷하고 sample size가 크고, response rate가 비슷하고, 모든 학생이 같은 학교에 다니므로 수용가능하다. 모든 학생이 비슷한 패턴으로 변할 것이라는 가정이 가능하다.

One notable limitation of this study is the use of a cross-sectional design,while a longitudinal design would have been more appropriate. Still, a cross-sectional design is deemed acceptable since the groups are comparable in age and gender, the sample size is large, the response rate is comparable, and all students attended the same medical school and in the same curriculum. It is therefore appropriate to assume that all students will change in a similar way (William & Darity2008). 

Med Teach. 2015 Aug 27:1-9. [Epub ahead of print]

Self-regulated learning and academic performance in medical education.

프로페셔널리즘의 변화: 깨진 거울에 비친 모습 (Med Teach, 2015)

The changing face of professionalism: Reflections in a cracked mirror

TREVOR GIBBS

AMEE, UK

가변성은 삶의 법칙이다. 어떤 두 얼굴도 서로 같지 않고, 어떤 두 신체도 같지 않다. 또한 어떤 두 사람도 우리가 질병이라 부르는 비정상적인 조건에서 동일하게 반응하거나 행동하지 않는다. William Osler

Variability is the law of life, and as no two faces are the same, so no two bodies are alike, and no two individuals react alike and behave alike under the abnormal conditions which we know as disease. William Osler (1949–1919)

William Osler가 한 이 말은 의학교육과 학생들에게도 동등하게 적용될 수 있다. 두 명의 환자가 절대로 동일하지 않은 것처럼, 두 명의 학생, 두 가지의 교육 및 직업환경도 절대로 동일하지 않다.

Much of what the renowned clinician, William Osler, was quoted to have said regarding clinical medicine can be equally applied to medical education and its students; just as two patients are not likened the same, neither are two students nor two educational or vocational environments.

Koehn과 Swick은 모든 의과대학생들이 문화적역량을 갖춰야 할 뿐만 아니라 초국가적인 역량 역시 갖춰야 한다고 주장했다. 즉, 국가간 관계에서 유도된 기술, 여러 문화를 아우르는 심리학과 여러 문화권 사이의 의사소통능력 등을 갖추어서 변화하는 사회에 적용시킬 수 있어야 한다는 것이다. 그들은 우리가 가르치는 것(문화적 역량)과 문화에 따라서 역량이 얼마나 다양하게 바뀔 수 있는가(문화 이외의 다른 요인들에 의해서 영향을 받은 역량)의 차이를 transnational competency라 명명하였다.

Koehn and Swick (2006) described the need for all medical students to not only become culturally competent but also to become transnation- ally competent as well; the need to produce students having a comprehensive set of skills, derived from international relations, cross-cultural psychology and intercultural commu- nication, equipping themto work in and adapt to our changing society. They describe the difference between what we teach – cultural competence and how various competencies change as a result of culture, with what happens when other factors other than culture affect those competencies – transnational competency.

프로페서녈리즘은 새로운 것이 아니며 히포크라테스 시절부터 존재해왔다. 이 당시에는 판단, 윤리, 특히 지혜가 핵심적 특징이었고, 매우 강한 실증주의적 관점과 함께 발전해왔다.

Professionalism is not new; with its origins firmly based on the early Hippocratic era, when judgement, ethics and has particularly wisdom were its key features, which it developed with a strongly positivist stance.

전문직업성에 관한 더 포괄적인 정의는 Epstein과 Hundert가 한 것으로 아래와 같다.

A more expansive definition of professionalism is Hundert provided by Epstein and (2002): ‘‘Professional competence is the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served’’. 

Hilton과 Southgate는 현대사회에서의 의료 전문직업성은 환자를 돌보는 것에 있어서 필요한 광범위한 자질을 포괄하고 있으며, 전통적인 의미의 전문성을 함양하고, 자율성을 가지고, 자기조절을 하는 것 이상이다"라고 했으며, 전문직업성이 역동성을 가지면서, 사회의 변화에 따라 적용해나갈 수 있는 것임을 암시했다.

Hilton and Southgate (2007, p. 265) recognized that ‘‘Medical the Professionalism in today’s society requires exhibition of a range of qualities deployed in the service of patients, rather than more traditionally defined aspects such as mastery, autonomy and self-regulation’’, alluding to the need for professionalism to be a dynamic entity, capable of adaptation to a changing world.

조금 더 일찍이, Schon은 진료에 대한 이론과 현실 사이의 이분성을 이야기하면서, "의사들이 연구에서 도출한 이론과 기술을 효과적으로 활용할 수 있는 높고 단단한 기반과, 온갖 상황이 혼란스럽고 난장판인 질퍽질퍽한 저지대"라고 묘사했다. 또한 우리가 가르치는 것과 우리가 노출되는 현실의 차이를 언급하였다. 우리가 명료하다고 가르치는 'learning organization'은 현실이라는 탁한 물 속에서 언제나 적용가능한 것은 아니다. 우리가 전문직업성에 대해서 가지고 있는 관점을 지지해주는 성찰적 거울은 우리가 실제로 발을 딛고 있는 현실에 의해서 깨어지고, 그 금(crack)으로 인해 상이 왜곡된다.

Somewhat earlier, it was Scho¨n who spoke of a dichotomy between our use of theory to inform practice and the reality of the real world, when he described the evocative image of: ‘‘...the high, hard ground where practitioners can make effective use of research-based theory and technique, and the swampy lowland where situations are confusing ‘‘messes’’ incapable of technical solution’’(Scho¨n 1983, p. 42). He too was describing the difference between what we teach and what we are exposed to in reality. What we see and teach in the clarity pervading the learning organization is not always applicable in the murky waters of the real world; what we see in the reflective mirror that support our views of profession- alism is frequently distorted by the cracks caused by the situation where we find ourselves in.

시뮬레이션 교육에서 방향이 제시된 자기조절학습 vs 교수자 지도의 학습(Med Educ, 2012)

Directed self-regulated learning versus instructor-regulated learning in simulation training

Ryan Brydges,1 Parvathy Nair,2 Irene Ma,3 David Shanks2 & Rose Hatala2

자기조절학습은 다음과 같이 정의된다. 많은 의료전문직 관련 기관들은 SRL을 비전과 미션에 포함시키고 있으며 이 때 SDL이나 평생학습과 같은 용어를 사용한다. SRL과 SDL이 여러 공통점이 있지만, 의학교육연구에서 이 둘은 구분되어 사용된다.

Self-regulated learning has been defined as a process involving ‘self-generated thoughts, feelings and actions that are planned and cyclically adapted to the attainment of personal goals’.15 Many medical professional agencies have included SRL in their visions and mission statements, using terms like ‘self- directed learning’ (SDL) and ‘lifelong learning’.16 Although SRL and SDL have many similarities,17 medical education researchers have used the terms differently: 

• SDL is often invoked when a learning environment is designed to promote autonomous learning (e.g. problem-based learning), whereas (학습환경이 자율적 학습을 촉진하도록 설계된 것)
• SRL is mentioned when the focus is on understanding the mechanisms of autonomous learning in order to identify how best to support learners when they engage in SRL.18 (자율적 학습의 메커니즘 이해에 초점을 두고 있으며, 이를 통해서 어떻게 SRL을 하는 학습자를 가장 잘 지원할 수 있는가를 알기 위한 것)

IRL과 DSRL을 비교하였다.

We compared this traditional approach of instructor-regu- lated learning (IRL) with what we call ‘directed self- regulated learning’ (DSRL).

절차 Procedure

참가자들은 baseline 설문지를 작성하였다.

Participants completed a baseline questionnaire on their previous experience with LP training (with both simulation and real patients) and reported their baseline confidence in performing LP on an 11-point Likert scale.

DSRL의 프로토콜 Protocol for directed self-regulated learning

IRL의 프로토콜 Protocol for instructor-regulated learning

네 명의 평가자가 global rating scale (GRS)와 checklist (CL)을 사용하여 pre, post, retention test를 평가하였다. 각 비디오에 대해서 두 명이 평가하였고, 공평하게 분배했다. GRS의 validity, reliability는 psychometric study로 supported 된다. 우리는 기존에 validity와 reliability가 확보된 CL을 가지고 그것의 변형된 버전을 만들었다. 일부 상관없는 문항을 삭제하였다. 비록 이러한 일부 문항의 삭제가 전문가 합의과정을 통해서 결정되었지만 validity를 다시 평가하지는 않았다. 대신 ICC를 계산하였다.

Four trained, blinded expert raters (authors RH, IM, PN, DS) used a global rating scale (GRS) (Appen- dix S1, online) and a procedural checklist (CL) to independently evaluate participants’ videotaped per- formances on the pre-test, post-test and retention test. Two raters evaluated each video; the rating load was distributed equally amongst the four raters. The concurrent validity, construct validity and reliability of the GRS are supported by a psychometric study on procedural skills.24 We created a modified version of a CL with demonstrated content validity and reli- ability,11 which consisted of 26 major and 44 minor actions. We removed some items fromthe original CL that we deemed as non-applicable to our simulation scenario, which shortened the overall CL, but main- tained an emphasis on major actions; specifically, we included 21 major actions and 14 minor actions (Appendix S2). Although the removal of items was based on expert consensus, we did not re-evaluate the validity of our modified CL; however, we did assess inter-rater reliability using the intra-class correlation coefficient (ICC).

평가자 훈련 Rater training

Training protocol

Prior to rating the videotaped performances, the four raters engaged in training. The training protocol involved the rating of three randomly selected videos, a meeting to discuss disagreements, the rating of another three randomly selected videos and a meet- ing to discuss these, followed by a final rating and discussion of four randomly chosen videos.

여기서 사용한 10개의 비디오는 ICC에서 계산시 포함하지 않았음.

We did not use the ratings of those 10 videos in the ICC reliability calculation, but did include them in the remaining analyses. During the training, the four raters developed a shared understanding of how to use the GRS and CL. One significant training outcome was the raters’ mutual definition of resident competence as ‘a resident being capable of performing LP in the clinical context with direct supervision’.

통계 분석

For these two performance variables, we assessed group differences between the DSRL and IRL groups on the pre-test and post-test, and in the pre-test, post- test and retention test scores using two separate repeated-measures analyses of variance (ANOVAs) with test as the within-subjects factor and group as the between-subjects factor. We also computed Pearson correlation coefficients to assess the relationship between participants’ self-reported confidence and the performance variables at the pre- and post-tests. Finally, we used separate independent samples t-tests to examine group differences in participants’ self- reported LP experience, total number of LPs per- formed or observed, total time with an instructor, and total practice time. We used Tukey’s honestly signif- icant differences (HSD) test and calculated the bias- corrected Hedges’ g effect size for the appropriate post hoc comparisons. All data are reported as mean ± standard error (SE) and the alpha level was set at p < 0.05.

본 연구에 대한 설명 중 하나는, 교수자가 연습 중에 제공한 피드백이 학습에 큰 도움이 된다고 여기는 것과 관련되어있다. 그러나 안타깝게도 motor learning에 관한 수십년의 연구에서 동시적 피드백(concurrent feedback)이 오히려 장기 학습에 해롭다는 결과가 나왔다. 전형적인 형태는, 연습 중에는 그것이 도움이 되는 것으로 보이지만, 피드백이 불가능한 상황인 retention 평가에서는 급격한 하락이다.

One explanation for our findings is associated with the provision by instructors of feedback during practice, which is commonplace in many domains and is considered a potent aid to learning. Unfortu- nately, decades of motor learning research have shown that concurrent feedback can be detrimental to long-term learning.26 Typically, findings show a benefit during practice, resulting in rapid skill acquisition, followed by poor performance on reten- tion tests when the feedback is unavailable.27

연습하는 중간에 제공되는 동시적 피드백의 촉진적 특성은 무시하기 어려우며 IRL이 의학에서 왜 그렇게 널리 쓰이는지를 설명해준다. 그러나 이러한 직관에도 불구하고 우리의 결과는 performance-learning paradox를 지지하는 근거를 하나 더 추가했을 뿐이다. (즉각적인 이득이 나타나나, 이것이 장기적인 학습을 반영하지는 못한다는 것) 이러한 결과가 직관에는 반하는 것일 지 모르나, 수많은 연구들이 장기적 학습에 있어서 '바람직한 어려움'의 활용을 지지한다.

The facilitating nature of concurrent feedback during practice is difficult to ignore and probably explains how pervasive IRL is in medicine. Despite these intuitions, our results add to evidence supporting the performance–learning paradox, which refers to the common finding that immediate performance bene- fits (as demonstrated in post-test scores) do not always reflect long-term learning (as demonstrated in retention test scores).28 Counterintuitive as this paradox may be, a wealth of evidence supports the use of ‘desirable difficulties’ to depress immediate performance and improve long-term learning.28,29

두 번째 개념틀은 자기모니터링이다. Eva와 Regehr는 자기모니터링을 '자신이 특정 상황에서 필요한 기술과 지식을 유지할 수 있는 가능성에 대한 순간순간의 판단'이라고 정의한다.

A second conceptual framework for interpreting the findings comes fromresearch on self-monitoring. Eva and Regehr31 defined self-monitoring as ‘a moment- by-moment awareness of the likelihood that one maintains the skill or knowledge to act in a particular situation’.

본 연구에서 DSRL그룹은 여러 시뮬레이터 모델 사이에서 점차 발전해나가며 자신의 능력과 준비도를 자기모니터링할 수 있다.

In the present study, the DSRL group’s opportunity to progress between simulator models may have led participants to self-monitor their ability and ‘readiness’ frequently.

교수들이 사용할 수 있는 시간은 제한적이고 이것이 흔히 simulation-based training 도입의 장애로 여겨진다. 연구자들은 교육효율성을 표준 성과 변인으로 고려해볼 필요가 있다.

Given that limitedfaculty staff time is a frequently cited barrier to the implementation of simulation-based training,33 researchers may benefit fromconsidering training efficiency (e.g. use of faculty staff time) as a standard outcome variable.

그러나 총 hands-on simulator time으로 계산하면 이 주장은 전혀 반대가 된다. (DSRL에서 hands-on time이 훨씬 많았음)

However, when we consider the total hands-on simu- lator time, the argument is reversed.

특히 DSRL을 위해서는 더 많은 자원이 필요하고, 어떻게 이러한 자원 필요량의 균형을 맞출 것인지가 중요한 문제일 것이다.

Further, more resources (e.g. LP trays and equipment) were needed for the DSRL than the IRL group. Determining how to balance these resource concerns represents a major challenge for simulation researchers.

Med Educ. 2012 Jul;46(7):648-56. doi: 10.1111/j.1365-2923.2012.04268.x.

Directed self-regulated learning versus instructor-regulated learning in simulation training.

층층이 쌓인... 복잡한 학습환경에서의 자기조절 (Med Educ, 2012)

Layers within layers … self-regulation in a complex learning environment

Rukhsana W. Zuberi

학습자가 동등한 지위의, 협력적 목표를 지닌, 그룹 내 접촉에 호의적인 사회적 규범을 지닌 환경에 지속적으로 노출되는 것이 중요함을 강조한다.

The articles1–3 correctly emphasise the value of repeated contact among individuals in a context in which equal status, cooperative goals and social norms favour inter-group contact."

또한 facilitation, guidance, support and role-modelling 가 중요하다.

The papers also repeat- edly define the importance of facilitation, guidance, support and role-modelling by faculty members."

롤모델의 중요성은 아무리 강조해도 지나치지 않으며, 롤모델은 스스로 무의식적, 의식적 편견을 극복하기 위해 노력해야 한다.

The importance of role models cannot be overstated and, there- fore, the role models themselves must explore and overcome their own unconscious and conscious biases."

세 개의 논문이 행동을 바꾸기 위한 교육전략을 제시했지만 거기에 깔린 이론을 확인해 볼 필요가 있다. 예컨대, 교육방법은 자기효능감이 발달함에 따라서 조절해나가야 하며, 학습자의 노력에 의해 관리할 수 있는 과제를 주어야 한다. 만약에 너무 위압적이면 학생은 자신감을 상실하고 비난하게 될 것이며, 너무 사소하면 학생은 성취감을 느끼지 못할 것이다. social cognitive theory에서 제시한 세 가지의 자기-반응적 요인은 어떻게 개개인이 자신에게 요구되는 수준과 현재 수행능력의 차이를 다루는지 예측하게 해준다.

(i) perceived self-efficacy to achieve the standards; 

(ii) behavioural self-reaction to sub- standard performance, and 

(iii) readjustment of personal stan- dards." 

For this reason, although the three articles1–3 published in this issue did survey educational strategies to change behaviour, the theories underpinning these remain to be explored. For example, it is broadly recognised that pedagogy must be moderated by the development of self-efficacy4,5 and by giving tasks that can, with effort, be managed by the learner. If the opportunities for practice are overwhelming, stu- dents lose confidence and embark on the blame game, but if the tasks are trivial, they gain no sense of achievement. Three ‘self-reactive factors’ in the social cognitive theory of Bandura and Cervone6 predict how an individual will handle the discrepancy detected between his or her performance and expected standards. These factors are: (i) perceived self-effi- cacy to achieve the standards; (ii) behavioural self-reaction to sub- standard performance, and (iii) readjustment of personal stan- dards."

Grain-size 분석과 관련해서, Bandura는 자기조절은 그것의 하위 기능은 자기모니터링으로부터 시작된다고 했다. 자기모니터링은 자기진단, 자기동기부여, 변화를 필요로 하는 행동과의 시간적 근접성, 변화에 대한 동기, 어떤 행동이 자신이 속한 문화나 기관으로부터 어느 정도 가치를 인정받는지 등을 포함한다.

In support of the grain-size analysis mentioned in the paper on self-regulation,1 Bandura7 has stated that self-regulation is initi- ated by the self-monitoring sub- function. Self-monitoring includes self-diagnosis, self-motivation, tem- poral proximity to the behaviour that requires change (the impor- tance of recency), motivation to change, and an understanding of whether the behaviour to be acquired is valued or not by the institution or the culture (in terms of the hidden or informal curricu- lum)."

Med Educ. 2012 Jan;46(1):7-8. doi: 10.1111/j.1365-2923.2011.04169.x.

Layers within layers … self-regulation in a complex learning environment.

해부실습: 가이드가 적고 자기주도적인 학습 vs 엄격한 가이드와 스테이션 기반 학습의 비교 (Anat Sci Educ, 2012)

Loosely-guided, self-directed learning versus strictly-guided, station-based learning in gross anatomy laboratory sessions.

Kooloos JG , de Waal Malefijt MC, Ruiter DJ, Vorstenbosch MA.

연구질문

(1) do strictly-­guided gross anatomy laboratory sessions lead to higher learning gains than loosely­-guided experiences? and 

(2) are there differences in the recall of anatomical knowledge between students who undergo the two types of laboratory sessions after weeks and months? "

연구방법

The design was a randomized controlled trial.

해부학 지식은 12개 구조물의 이름을 실습 직후, 1주 후, 5주 후, 8달 후에 걸쳐 측정함으로써 진행되었다.

The recall of anatomical knowledge was measured by written reproduction of 12 anatomical names at four points in time: immediately after the laboratory experience, then one week, five weeks, and eight months later. "

SGG가 LGG보다 네 차례 모두에서 더 높은 점수를 받았다.

The strictly-­guided group scored higher than the loosely-­guided group at all time ­points. Repeated ANOVA showed no interaction between the results of the two types of laboratory sessions (P = 0.121) and a significant between-­subject effect (P ≤ 0.001). Therefore, levels of anatomical knowledge retrieved were significantly higher for the strictly-­guided group than for the loosely-­guided group at all times"

Anat Sci Educ. 2012 Nov-Dec;5(6):340-6. doi: 10.1002/ase.1293. Epub 2012 May 31.

Loosely-guided, self-directed learning versus strictly-guided, station-based learning in gross anatomy laboratorysessions.

문화에 따라 달라지는 SDL (Med Educ, 2012)

The culturally sculpted self in self-directed learning

Gary Poole

SDL에서 'self'라는 단어는 학습에 대한 책임과 그 성과에 대한 책임의 위치(locus)를 의미한다.

The word ‘self’ in self-directed learning (SDL) indicates a locus for both the responsibilities related to learning and the outcomes of that learning."

문화권에 따라서 학습에 대한 책임은 개인에게 있을 수도 있고 집단에 존재할 수도 있다.

responsibility may be located in the individual or in the collective."

학생들은 아래와 같은 질문들을 던져봐야 한다.

Students must think about themselves and ask some important questions. Can I effectively evaluate my own performance? Do I believe I can learn independently? Am I independently motivated to learn? Can I stay organised?

이들 질문에 대답을 하기 위해서는 일종의 'constructive narcissism'이 필요하다 (건설적 자아도취). 학생은 자신에 대해서 생각하는 것이 '이기적'인 것이 아니라, 필요한 것이라고 생각해야 한다. 실제로 자신을 성찰의 대상으로 포함하는 자기성찰 과정은 성공의 필수적 요소로 여겨진다. 이러한 종류의 자기성찰의 특성은 문화와 개인에 따라 달라진다.

The effort required to answer these questions calls for a kind of ‘constructive narcissism’. One must believe that thinking about the self is not ‘self-ish’, but necessary. Indeed, the process of reflection that includes the self as an object of that reflection is considered central to successful SDL.3 The nature of this sort of reflection will vary across cultures and individuals.4,5"

더 나아가 우리가 스스로에 대해서 생각하는 방식은 자기 자신 내에서의 비교와 다른 사람과의 비교를 모두 필요로 한다. 아시아와 중동의 학생들은 서양 학생들보다 더 경쟁적이라는 것을 보여주는데, 여기서 '경쟁적'이라는 단어는 자기자신을 상대방과 비교함으로써 정의내리는 경향을 말한다. 경쟁적인 학생들은 외부의, 상대적인 데이터 (시험성적과 같은) 정보를 필요로 한다. PBL에서 그러한 학생들은 지속적으로 자신의 지식을 다른 사람들의 지식과 비교한다.

Furthermore, the way we think about ourselves may well involve the making of comparisons, both within one’s self and between self and others. The authors’ data indicate that students in Asian and Middle Eastern contexts are more competitive in nature than their Western counterparts.1 I would argue that ‘competitive’ refers to a tendency to define one’s self in comparative terms. Competitive students require external, comparative data from things like examinations. In problem-based learning (PBL) settings, such students may continually compare their knowl- edge with that of other group members."

학생들에게 SDL을 원하느냐는 질문에, SDL이 PBL부터 자기주도적 연구 프로젝트까지 다양하고 일반적인데 서양의 의과대학에서 답은 당연히 '그렇다'일 것이다. 그러나 Frambach는 서양의 이러한 가치가 모든 곳에서 유효하다는 가정은 조심해야 한다고 주장한다. 그들은 이렇게 말한다. '학생 중심, 문제 중심의 방식을 적용하는데 맞닥뜨리는 문화적, 맥락적 어려움과 싸우느니, 차라리 그 맥락에 맞는 대안을 새롭게 만들거나 찾아내는데 노력을 들이는 편이 현명하다.'

Western medical schools, in which SDL opportunities that range from PBL to self-directed scholarly projects are common, the answer would be yes. In their concluding remarks, however, Frambach et al.1 caution against the assumption that Western values can be uni- versally applied. They state: ‘Rather than taking on the cul- tural and contextual challenge of adopting student-centred, prob- lem-based methods, it might be wiser for medical educationalists to rise to the challenge of exploring or creating alternatives that best fit their particular context.’1"

Frambach J, Driessen E, Chan L-C, van der Vleuten CMP. Rethinking the globalisation of problem-based learning: how culture challenges self-directed learning. Med Educ 2012;46:738–47."

The culturally sculpted self in self-directed learning.

자기조절학습을 다룬 의학교육연구에 대한 자성적 분석 (Med Educ, 2012)

A reflective analysis of medical education research on self-regulation in learning and practice

Ryan Brydges & Deborah Butler

의학에서 자기조절에 관한 연구는 그 역사가 길다.

The study of self-regulation in medicine has a long and rich history.

자기조절에 대한 연구는 의료전문직으로서 필요한 요건의 한 가지로서 관심이 집중되어왔고, 의료전문직이란 자기조절을 하는 전문가로서 지속적인 전문성 개발을 적극적으로 이뤄내서 최소한의 역량을 유지할 수 있어야 하기 때문이다. 유사하게 의학을 배워나가는 학습자들도 의학교육에 유능한, 자기조절적 학습자로서 참여할 것이 기대된다.

Correspondingly, attention has also focused on self-regulation as a requirement of medical professionals, who as part of a self- regulating profession, are expected to identify and willingly engage in ongoing professional develop- ment activities that serve to maintain a minimum level of learning and competence.2,3 Similarly, in preparation for joining medical practice, medical trainees are also expected to engage in medical education as capable, self-regulating learners.4

SRL은 '스스로 생각, 감정, 행동을 만들어서 자신의 목표를 달성할 수 있도록 계획하고 주기적으로 적용해나가는 것'으로 정의되어왔다.

Self-regulated learning (SRL) has been defined classically as: ‘self-generated thoughts, feelings and actions that are planned and cyclically adapted to the attainment of personal goals.’6

임상실습을 도는 학생을 상상해보자.

Firstly, let us consider a medical student engaged in her first clinical clerkship.

학생은 이 기간동안 주로 그전 학습환경에서 사용해온 학습전략을 지속적으로 활용하나, 여러 근거들을 종합하면 임상환경에서 그러한 전략들은 대체로 효과적이지 못하다.

Evidence suggests that during this transition she will persist in using learning strategies (e.g. memorisation) that may have worked in previous settings, yet are not as effective in the clinical environment.8,9

이 학생은 도움을 요청해야 할 때가 언제인지 적절한 판단을 내림으로써 유능하게 보일 수도 있지만, 동시에 환자의 안전도 담보해야 한다. 동시에 임상경험을 통해서 학습 진행을 관리해야 한다. 학생은 학습, 봉사, 의료의 우선순위가 서로 상충한다는 것을 곧 깨닫게되며, 자신의 시간관리를 스스로 해야한다.

She must make careful judgements about when to ask for assistance so that she can appear compe- tent, but preserve patient safety. Simultaneously, she must manage her learning about and through her practice experiences. She quickly realises that learn- ing, service and patient care can be conflicting priorities, and that she must manage her time largely on her own.9

로테이션 시스템에서 학생은 길을 잃은 느낌을 받거나 그 구조의 불합리성을 비난하고, 임상에서의 위계질서는 병원에서 있느니 책 한글자라도 더 보는 것이 배우는 것이 많겠다는 생각을 하게 한다.

She may start to feel lost and blame the work structure in the form of the rotational system and clinical hierarchy for making her feel as if she learns more from reading a textbook than she does from her time at the hospital.9

둘째로, 임상의사는 지속적으로 전문성을 함양하여 역량을 유지하고 최신 지식을 유지하면서 자기조절을 하게 된다. 그러나 자기조절에 대한 주위의 기대치는 임상 교사와 같은 그들이 맡은 또 다른 역할에서도 드러난다.

Secondly, practising clinicians self-regulate when they engage in ongoing professional learning to maintain competencies and stay up to date. However, expecta- tions for self-regulation also emerge within other of their roles, such as that of the clinical teacher.11,12

학계든 지역사회든 모든 환경에서 의사는 교수자로서 자기조절(피교육자에 대한 교육을 어떻게 할 것인가)와 평생학습에 대한 자기조절(어떤 참고자료를 찾아봐야 하며 언제 조언을 구해야 하는가)의 균형을 유지해야 한다.

academic or community care setting. In all settings, he must balance self-regulating his practice as teacher (e.g. in how he educates his trainees and patients) and self-regulating his lifelong learning (e.g. in how he makes decisions about what to look up and when to consult with colleagues).

이 두 가지 사례는 우리가 의학교육에서 자기조절에 대해 생각하는 방식을 묘사한다.

These two examples illustrate current ways of think- ing about self-regulation in medical education.

어떤 영향력은 학습자 외부에 있고, 어떤 영향력은 학습자에 내재되어 있다.

Some influences are external to learners, Other influences can be associated with a learner,

이러한 사례를 기반으로 보면, 의학교육에서 자기조절이란 다양한 내적, 외적 영향력에 주의를 기울일 것이 요구된다.

Given these examples, it would seem that understanding self-regulation in medical educa- tion requires attending to a range of internal and external influences.5

Figure 1 A model of self-regulation (adapted from Butler et al.5; see also Cartier and Butler16)

개개인이 학습환경에 가져오는 것

What individuals bring to the learning context

그러나 개개인이 자기조절에 대해 접근하는 방식은 학습환경을 곧바로 반영하는 것은 아니다. 개개인은 다양한 오랜 기간에 걸쳐서 발전되어온 지식, 신념, 행동, 역사, 경험 등을 특정 환경에 가져온다. 성공적인 학부 의과대학생은 의과대학에 처음 입학했을 때는 자신감에 넘쳤을 수 있지만 머지 않아 자기주도적 교육과정의 낯선 요구에 겁을 먹을 것이다.

However, an individual’s approach to self-regulationis not a direct reflection of context. Individuals bring to contexts a variety of knowledge, beliefs andemotions that have developed over time through their history and experiences and that emerge in particular settings. Successful undergraduate learn- ers, for example, might be confident when enteringmedical school, but then may be rattled by the unfamiliar demands of a self-directed curriculum.19

자기조절 관련 행동의 사이클

A cycle of self-regulation in action

역사적, 사회적, 문화적 세팅에서의 자기조절

Self-regulation in historically, socially and culturally situated settings

This research underlines the importance of considering how observed patterns in 

• self-regulation (e.g. the pattern demonstrated by a medical trainee who fails to ask for help when he needs it to ensure patient safety) 
• emerge from 
• complex interactions between cultural norms (e.g. expectations to act independently), 
• pedagogical practices (e.g. feedback) and 
• individuals’ interpretations and aspirations (e.g. the desire to assume the identity of a doctor).

SRL을 요구하거나 촉진하는 학습환경

Learning environments that demand or foster SRL

또 다른 연구에서 어떻게 교육환경이 자기조절을 촉진하는지 알아보았다. 

In another group of studies, medical education researchers have focused on how pedagogical envi- ronments might foster self-regulation.26 An impor- tant example can be found in the rich history of articles that invoke the term ‘self-directed learning’ (SDL). Mapped on to our theoretical framework, one contribution of this line of research is that it attends specifically to how environments can be designed to provide opportunities for and expect self-regula- tion,13 such as by using tools like PBL and life-long learning modules.27 Evidence to support such SDL initiatives is encouraging, if still in its early stages.

White의 연구를 보면, 학생들에게 SRL을 요구하는 것은 궁극적으로는 도움이 된다. SRL은 SR을 기대하거나 가능하게 하는 환경에서부터 촉진될 뿐만 아니라, 그러한 환경에서 자기조절을 지원하고, 개개인에게 어떻게 SRL을 할 수 있는지 알려주는 것으로부터도 촉진된다.

As we considered the study by White19 (and other similar studies) in relation to emerging strands of research, our analysis suggested that, although it is ultimately beneficial for students to experience demands for self-regulation, either in an early PBL experience or when transitioning to clerkship, students in both groups may have been similarly disoriented and challenged at the moment when learning expectations changed (i.e. in PBL in Year 1 or in the clerkship in Year 3), and may have benefited from support to navigate those changes.13 This obser- vation is grounded in evidence which suggests that self-regulation is fostered not only by establishing environments that afford or expect self-regulation, but also by supporting self-regulation in those contexts and by assisting individuals to learn how to self-regulate their learning. Medical educators are starting to study self-regulation from this perspective by drawing on the seminal work of Irby12,28 to consider how to manage trainees’ progression toward independent learning and practice.10,11

self-directed learning에서 중요한 것은 그것을 너무 문자 그대로 해석하지 않는 것이다.

A key implication is that the term ‘self-directed learning’ should not be interpreted too literally.31

어떻게 자기조절이 co-regulated practice의 형태로 guide될 수 있으며, support될 수 있는지를 강조하고자 한다.

Thus, we join others in calling for greater emphasis on how self-regulation can be guided and supported as a form of co-regulated practice (i.e. practice that is shaped by context and by others).13,29–32

개개인이 학습환경에 가져오는 것

What individuals bring to the learning context

의학교육연구는 어떻게 이전 지식, 신념, 감정이 수행능력을 좌우하며, 왜 SRL이 항상 이상적으로 작동하지 않는지에 대해 설명해주었다.

Medical education research has examined how prior knowledge, beliefs and emotions mediate perfor- mance and can account in part for why self-regulation may not always unfold in an ideal manner.

종합하면, 이전 경험에서의 지식과 헌신이 새로운 환경과 상호작용하여 self-regulation에 영향을 준다.

Collectively, these studies show that knowledge and commitments from prior experience interact with context (e.g. clarity of expectations) to influence self-regulation.8,32

자기조절행동의 사이클

A cycle of self-regulation in action

최근까지 자기평가에 대한 연구는 학습자들이 얼마나 자기 지식의 한계를 잘 판단하며 그 정보를 가지고 전문성개발의 가이드로 삼는지에 대한 것이었다.

Until recently, most self-assessment research has focused on measuring how well trainees or practitioners can judge the extent or limits of their knowledge and use that information to guide their professional development.37

그러나 최근의 연구는 분석의 입자 크기에 보다 집중하고 있다. 

However, recent research has paid closer attention to the grain size of analysis,39 which has important implications for understanding the potential for trainees and doctors to adequately self- assess.

자기평가의 이전 연구는 개개인이 자신들의 경험을 축적하여 총괄적 자기평가를 할 수 있다는 가정에 이뤄졌지만, 보다 최근의 연구는 어떻게 주어진 SRL 사이클에서 실제 수행 중에 self-monitor 혹은 self-assess를 할 것인가에 집중되어 있다. self-monitor에 대한 최근의 두 가지 연구는 특정 순간에 self-monitor를 하는 것이 global self-assess보다 더 정확하고 민감하다는 것을 보여준다.

That is, previous study of self-assessment assumed that individuals are capable of aggregating across experiences to generate global self-assessments which will spur professional learning. By contrast, more recent work has focused on how an individual self-monitors or self-assesses within a situated SRL cycle during practice (Fig. 1). Two recent studies of the self-monitoring process suggest that individuals are more accurate and sensitive when self-monitoring in the moment than they are in making global self- assessments.35,37

의학교육의 복잡성

Complexity of medical education

첫째로, 우리는 자기조절에 대한 분석이 학습과 진료의 복잡성을 고려해야 한다고 본다.

Firstly, we recommend that any analysis of self- regulation in medical education recognise and take into account the complexity of learning and practice.

자기조절에 대한 개념

Conceptualising self-regulation

앞에서 제시한 프레임워크르 사용하는 것이 다양한 SRL에 대한 연구를 조화롭게 엮어주는 것을 확인했다.

We found that a benefit of using our integrative framework in our analysis was that it enabled us to connect different lines of research on self-regulation in a coordinated and coherent manner.

입자 크기에 대한 관점의 유용성

Grain size: a useful perspective

자기조절에 대한 연구로부터 우리는 연구자들이 grain size에 관심을 두어야 함을 주장하고자 한다. grain size란 연구자가 연구를 위해 선택하는 분석 혹은 디테일의 수준이다.

Building from research in self-assessment, we recom- mend that researchers attend to the grain size of analyses of self-regulating processes. Grain size refers to the level of detail or analysis one selects for study. 

• For example, a fine-grained analysis might focus on instances of self-regulating processes as they play out in context (such as when a medical trainee interacting with a patient with diabetes decides that his expertise is exceeded and that he needs help from a supervisor), 
• whereas a more global analysis might ask for a generalised assessment of knowledge relevant to practice (such as when the same trainee rates how much he knows about diabetes to guide his learning).

자기조절에 대한 지지

Supporting self-regulation

두 가지 주요한 가정에서 탈피해야 한다. 한 가지는 독립적으로 완수할 수 있는 학습활동을 설계하는 것이 내용 영역의 학습과 자기조절의 개발 모두를 달성할 수 있다는 것이며, 둘째는 자기조절이라는 것이 온전히 학습자 내부에서 진행되는 것이고 따라서 교수는 거의 할 일이 없다는 생각이다.

Our analysis of research suggests two major assump- tions to be avoided in the study of self-regulation in medical education. The first is the assumption that designing an activity so that it can be completed independently is sufficient to promote both learning of a content domain and the development of self-regulation. The second is the assumption that self- regulation is an activity conducted entirely within the learner and, consequently, that faculty members play little or no role in supporting self-regulation.13

자기조절능력의 향상은 내용 전문가로서 발전하는 것과 함께, 그리고 그 한 부분으로서 도달할 수 있다. 

individual and that content knowledge and effective self-regulation in a given area can develop progressively and in tandem (i.e. bootstrapping can occur).

45 Eva KW, Regehr G. Exploring the divergence between self-assessment and self-monitoring. Adv Health Sci Educ Theory Pract. 2011;16(3):311–29.

Med Educ. 2012 Jan;46(1):71-9. doi: 10.1111/j.1365-2923.2011.04100.x.

A reflective analysis of medical education research on self-regulation in learning and practice.

의대 과정동안 자기주도적학습 능력이 향상될까 줄어들까? (Acad Med, 2013)

Does Medical Training Promote or Deter Self-Directed Learning? A Longitudinal Mixed-Methods Study

Kalyani Premkumar, MBBS, MD, MSc (Med Ed), PhD, Punam Pahwa, PhD,

Ankona Banerjee, MSc, Kellen Baptiste, MD, Hitesh Bhatt, MSc, and Hyun J. Lim, PhD

Campbell 등은 SDL을 Maslow가 '자기실현적 개인'이라고 칭한 고도로 주도적인 자기 학습자와 같이 학습 프로그램에 참여하는 것에 있어서 스스로 주도하고, 스스로 기획하는 행동이라 정의했다. Hammond와 Collins는 SDL을 학습자가 주도권을 쥐고 다른 사람의 지원과 협력하에 진행되는 프로세스라고 묘사했다. 그러나 가장 잘 정의된 것은 Knowles가 개개인이 이니셔티브를 쥐고, 다른 사람의 도움이 있거나 없는 상황에서 스스로의 학습요구를 진단하고, 학습목표를 설정하고, 인적 물적 학습자원을 찾아서 적절한 학습전략을 선택 및 도입하고 스스로의 학습결과를 평가하는 것이다.

Campbell et al1 define SDL as behaviors that range from participation in programmed learning to the self- initiated, self-planned activities of such highly directed self-learners as Maslow’s self-actualizing individuals.2,3 Hammond and Collins4 describe SDL as a process in which learners take the initiative, with the support and collaboration of others. But we believe the concept of SDL is best captured by Knowles5 as a process in which individuals take the initiative, with or without the help of others, in diagnosing their learning needs, formulating learning goals, identifying human and material resources for learning, choosing and implementing appropriate learning strategies, and evaluating learning outcomes."

왜 의학에서 SDL이 중요한가? Why is SDL important in medicine?"

SDL은 의학 분야에서 매우 강조되어왔는데, 예컨대 CanMEDS에서 scholar role을 보면, 의사들이 일생에 걸친 자기성찰적 학습을 해야 한다. 또한 ABMS와 WFME는 SDL을 의학교육에서 반드시 평가해야 하는 특징으로 보았다.

SDL has been increasingly emphasized in the medical field. For instance, the scholar role of the CanMEDS 2005 Physician Competency Framework emphasizes SDL by requiring physicians to demonstrate a lifelong commitment to reflective learning,9 and both the American Board of Medical Specialties and the World Federation for Medical Education include SDL as a characteristic that should be evaluated during medical education.10,11"

ACGME 역시 practice-based learning and improvement를 여섯 개의 핵심 역량 중 하나로 보았으며, 추가적으로 CME 역시 의사들이 자기주도적으로 스스로의 학습요구, 목표, 학습활동 선택 등을 잘 할 수 있다는 가정하에 이뤄지는 것이다.

The Accreditation Council for Graduate Medical Education, too, identifies practice-based learning (a form of SDL) and improvement as one of its six core competencies.12 In addition, continuing medical education for physicians is based on the assumption that physicians are self-directed leaders who can accurately predict their own learning needs, set goals, engage in appropriate learning activities, and regularly and accurately assess the outcomes.13,14"

SDL의 역량 

Competencies of SDL"

SDL의 역량은 여러 사람이 묘사했는데, 학습 격차에 대한 이해, 자신과 타인에 대한 평가, 성찰, 정보 관리, 비판적 사고, 비판적 평가 등이 있다.

The competencies of SDL have been described by many.5,15,16 They include proficiency in assessment of learning gaps, evaluation of self and others, reflection, information management, critical thinking, and critical appraisal.15"

이런 것들이 '자율성'을 의미하는 것 같지만, SDL은 동료 및 선생님들과 서로 정보를 교환하는 것을 포함한다.

Although all these characteristics seem to indicate autonomy, SDL involves interaction with peers and teachers to exchange information.17"

SDL이 교육될 수 있는가? Can SDL be taught?"

SDL is often described as a continuum, present in all individuals to some extent, with those who are least self-directed being totally dependent on the teacher for learning.5,16 The other end of the spectrum is the totally self-directed learner, who independently determines what is to be learned, identifies the resources, solves problems, and evaluates. It should be noted that a person who is highly self- directed in a particular situation may be very much less self-directed in a new and unfamiliar context.18"

Grow는 SDL의 단계 모델을 만들면서, 학습자를 자기주도적 학습의 네 단계 중 하나에 있다고 보았다.

Grow,16 in his staged SDL model, describes the learner, at any given time or learning situation/ context, to be in one of four stages: 

• dependent (stage 1), 
• interested (stage 2), 
• involved (stage 3), and 
• self-directed (stage 4)."

교육자들은 학습자가 어느 단계에 있는가를 진단하고 더 높은 단계로 준비히켜주어야 한다.

Educators need to diagnose the learner’s stage of self-direction and prepare the learner to advance to higher stages."

According to Grow, although learning in a dependent mode goes against the principle of student-centered styles of teaching and adult learning, there is nothing demeaning or destructive in promoting temporarily dependent relationships as long as the purpose is to advance learners to higher stages."

SDLRS는 자기기입식 설문지로 다음의 구인을 평가한다.

The SDLRS consists of a self-report questionnaire of 58 questions and is one of the most common instruments used to assess SDL readiness.24 It measures eight constructs: 

• Openness to learning opportunities 

• Self-concept as an effective learner 

• Initiative and independence in learning 

• Informed acceptance of responsibility to one’s own learning 

• Love of learning 

• Creativity 

• Positive orientation to the future 

• Ability to use basic study and problem- solving skills"

질적분석 

Qualitative. 

연구 후반부에 인터뷰를 실시하였고, 여섯 명으로부터 반구조화 면접을 수행했다.

At the end of the study, all instructors in the medical school were invited to participate in interviews. Six instructors volunteered, and we conducted semistructured interviews with them."

녹취된 인터뷰와 포커스그룹은 세 명의 coder에 의해서 independently 분석되었고, 반복적인 토론을 통해서 합의달성

The recorded interviews and focus group discussions were transcribed. The transcriptions were independently examined for common themes by three coders (K.P., A.B., and another qualitative research expert). Repeated discussions were held until agreement among coders was attained."

입학 당시 SDL 준비도

SDL readiness and age at admission 

연령이 높아질수록 SDLRS 점수에 긍정적 효과가 있었다. 종합적으로, 더 나이를 먹은 학생이 더 점수가 높았다.

Age was considered to be a continuous variable in this study, and an increase in one year of age at admission had an overall positive impact on SDLRS scores. Overall, students who were older had significantly higher scores (P = .002) than did younger students."

SDL과 premedical university education 기간

SDL and years of premedical university education"

SDL 준비도의 변화

Changes in SDL readiness of medical students from admission to graduation."

입학 후 1년이 지났을 때 SDLRS점수에 유의미한 하락이 있었다. 입학시 점수보다 지속적으로 낮았다.

There was a significant drop (P < .001) in SDLRS scores in all cohorts one year after admission. The scores of all cohorts continued to be significantly lower than that at admission throughouttraining and at graduation."

SDL을 촉진하는 현 교육과정 내의 활동들

Activities in the current curriculum that promote SDL "

SDL에 배정된 시간을 중요시하여, 교육과정의 한 부분으로서 1학년 때 매일 오후는 SDL로 할당되었다. 그러나 지침이나 관리 없이 시간만 주는 것은 SDL을 오히려 악화시켰다.

The instructors and students identified some of the activities within the current curriculum that promote SDL: The time allocated for SDL—especially in the first year—was considered valuable. As part of the curriculum, a few hours of every afternoon in year 1 is earmarked for SDL. However, it was felt that being giventime, without direction and monitoring, deterred SDL."

SDL 촉진/악화 요인

Factors that facilitate/deter SDL 

두 가지 주제가 드러났다. 하나는 학습 환경과 관련된 것이고 다른 하나는 평가에 관련된 것이다.

Two themes clearly emerged: one related to the learning culture and environment and the other related to assessment."

SDL이 교육과정 전체에 걸쳐서 주요한 주제가 되어야 한다. 한 두 개의 과목만으로는 안된다.

Instructors felt that SDL has to be a theme throughout the curriculum, not introduced in just one or two courses."

SDL은 guide process로 여겨졌으며 SDL의 기술들은 교육될 수 있다

SDL was thought to be a guided process, and skills in SDL had to be taught."

SDL이 학년이 올라가면서 낮아지는 것에 대해서 교사나 학생 모두 전혀 놀랍지 않다는 분위기였다.

Towards the end of the focus group/ interviews, students and instructors were told that preliminary findings of this study seemed to indicate that SDL readiness was decreasing with increasing years. Both groups were not surprised at the findings:"

SDL readiness and age. 

나이를 더 먹은 학생이 어린 학생보다 유의미하게 높았으며 이는 Reio와 Davis, Kell과 Van Deursen의 결과와 일치한다. (50세까지는 SDL이 향상됨) 학습에 대한 선호는 과거의 학습경험과 학습과정을 통제할 수 있다는 자신감에서 나온다.

Older students had significantly higher scores than younger students. This is consistent with the findings of Reio and Davis38 and Kell and Van Deursen41 and lends further support that SDL has a positive developmental trajectory until the 50s, consistent with SDL theory.42,43 The learning preference has been attributed to previous learning experience and confidence in controlling the learning process.44 "

SDL readiness and premedical training"

This finding is in contrast to that of Harvey et al,39 who found a significant positive trend in SDL scores (using SDLRS, the Oddi Continuing Learning Inventory [OCLI], and Ryan’s [ability] scores) associated with the highest level of premedical education achieved (undergraduate only, master’s, or doctoral)."

SDLRS scores with increased medical training."

SDL점수가 1학년 말에 유의미하게 하락했다.

Our findings indicate that SDL readiness scores decreased significantly at the end of one year."

이러한 연구 결과는 U of Toronto의 결과와도 비슷하다.

our findings are similar to the findings of researchers at the University of Toronto Faculty of Medicine, who did a cross- sectional study on first- and second-year medical students (N = 280). Of the three instruments that they used (SDLRS, OCLI, and Ryan), the scores obtained with Ryan’s instrument showed a decrease with more training.39"

이런 차이는 어떻게 설명가능할까?

How can these differences be explained?"

이러한 차이는 어떻게 설명할 수 있을까? Knowles는 학습을 교수주도과 자기주도를 양측에 둔 연속체로 묘사하였다(pedagogical - androgogical). 이 연속체는 학습자가 학습에 대하여 얼마나 통제권을 가지고 있으며, 학습목표를 달성하기 위하여 필요한 평가와 전략에 대해 얼마나 자유도를 가지는지에 의해 달라진다. 예컨대 특정한 학습 영역에서 자기주도적이 되기 위해서는 학습자는 특정 수준 이상의 지식을 가지고 있어야 한다. SDL에 대해 얼마나 준비가 되어있느냐는 학생마다 매우 다르고, 이는 학생마다 경험이 다르기 때문이다. 학생이 지식이 적을 때는 높은 수준의 구조화가 필요하다. SDL에 대한 준비도와 틀에 짜여진 교육시간에 대한 선호도와는 분명한 역의 상관관계가 있다. 학부 교육과정동안 학생들은 과도한 정보를 주입당하는데, 학생들은 그 시간동안 자기들의 스스로의 흥미에 대해 공부할 시간이 거의 없다. 따라서 의대 기간동안 SDLRS가 떨어지는 것이 놀라운 것은 아니다.

How can the differences in SDLRS scores with training be explained? Knowles5 describes learning as a continuum with teacher-directed (pedagogical) learning at one end and self-directed (androgogical) at the other. This continuum can be explained in terms of how much control the learner has over learning and the amount of freedom given to evaluate and implement strategies to achieve learning goals. In medicine, the learning environment tends to keep students in the pedagogical end of the spectrum. For instance, to be self-directed in a specific content area, a person must possess a certain level of knowledge. The readiness for SDL is variable in any given student population, as each student enters medicine with a different academic background. When students have a low level of knowledge, they prefer a high degree of structure. It has been shown that there is a definite inverse correlation between SDL readiness and student preference for structured teaching sessions.47,48 Throughout undergraduate training, students are overloaded with information. Moreover, the competencies required are well defined by regulatory bodies, and there is very little time for students to pursue their own interests. As our faculty note, it is therefore not surprising that SDLRS scores decrease with training."

Knowles에 따르면 학생과 교사 모두 SDL기반 교육과정 도입에 필요한 스킬을 갖출 필요가 있다.

According to Knowles,5 both teachers and students have to possess the skills necessary for the implementation of an SDL-based curriculum."

• role of facilitators"
• learning environment that is collaborative rather than competitive."
• diagnose learning needs"
• help learners diagnose their own needs.5"
• Training plays a key role"

학생과 교사의 준비가 부족한 것 외에 다른 SDL의 장애물로는 Shokar는 "전문성, 교육과정, 법, 조직 내규, 외부 제약, 시간 제약, 학습할 내용 등으로부터 오는 제한"이라고 했다.

Apart from lack of teacher and student preparation, other factors may serve as barriers to SDL. Shokar et al50 list some of these barriers as “restrictions imposed by professional, curricular, legal, and institutional requirements, statutory educational regulations, time constraints, and the need to ensure that specific content is covered.”"

모든 상황에 SDL이 적용가능한 것이 아님을 염두에 둘 필요가 있다. 학생이 과거 경험이 매우 적거나 학습의 초점이 과목 그 자체보다는 내용(구체적 학습목표)에 있을 때 SDL은 적절하지 않다. 학부의학교육은 - 특히 1학년은 - 거의 지식에 대한 기반이 약하고 따라서 학생들이 교사-의존적이 되는 것은 이해할 만 하다.

One must also remember that not all situations are applicable for SDL. SDL may not be appropriate in situations where the student is new or has very little previous experience of the subject and when the focus of learning is on the content (e.g., specific learning objectives) rather than the subject itself.47 In undergraduate medical training— especially in the first year—most students have very little foundational knowledge. It is therefore understandable that students are more teacher-dependent and require that the education program be more structured."

SDL에 영향을 주는 요인들이 복잡함을 고려할 때 많은 모델이 SDL을 대표한다. Garrison의 모델, Brocket과 Hiemstra의 모델, Candy의 모델.

Given the complexity of factors that influence SDL, a number of models have been proposed to represent SDL. 

• Garrison’s51 model focuses on three psychological constructs: self- monitoring (cognitive responsibility), self-management, and motivation (see Figure 2). 
• Self-monitoring refers to the ability of learners to monitor both their cognitive and metacognitive processes. 
• Self-management focuses on goal setting, use of resources, and external support for learning. 
• Motivation has two dimensions: entering and task motivation. 
• Entering motivation is what compels the learner to participate in the learning process, whereas 
• task motivation is what keeps the learner on task and persisting in the learning process. To promote SDL, each of these constructs needs to be addressed."

• Brocket and Hiemstra’s52 model of SDL (see Figure 2) focuses on learner control of responses to a situation even if there is no control over the circumstances; the model considers SDL and learner self-direction as two dimensions, with personal responsibility connecting the two. To facilitate SDL, focus on promoting personal responsibility is required."

• Candy’s53 SDL model (see Figure 2) illustrates two interacting laminated (layered) domains. One dimension relates to the amount of control within an institutional setting, with one end of the continuum showcasing teacher control and the other learner control. The second dimension relates to amount of control over informal learning: autodidaxy. In this model, one needs to help the organization and its teachers choose appropriate strategies based on the content and level of knowledge of students, and to facilitate movement of students along the continuum."

다갈래의 전략이 필요하다. 학생이 SDL궤도를 따르도록 도와줄 수 있는 네 가지 원칙을 요약하였음.

From the different SDL models it can be deduced that a multipronged approach has to be taken to promote SDL in students. Francom54 summarizes four principles instructors can use to help students move along the SDL trajectory: 

• Match the level of SDL required in learning activities to student readiness 

• Progress from teacher to student direction over time 

• Support the acquisition of subject matter knowledge and SDL skills together 

• Have adults practice SDL in the context of learning tasks"

학생이 학습에 대한 책임을 지게 하는 교육자의 여섯 가지 역할을 제시하였음

Hiemstra55 describes six foundational roles that instructors need to take on to enable students to adopt personal responsibility for their learning: 

• Content resource (sharing expertise and experiences using various forums) 

• Resource locator (locating and sharing various resources to meet student needs)"

• Interest stimulator (arranging for resources that maintain student interest in the subject, e.g., games, discussions; guest presentations) 

• Positive attitude generator (through positive reinforcement; prompt, useful feedback) 

• Creative and critical thinking stimulator (through study groups; journal writing; logs; simulation; role-play) 

• Evaluation stimulator (learner evaluation and promotion of self-evaluation)"

Figure 2 Three models of self-directed learning (SDL).51–53 See the text for a discussion of these models. Used with permission."

Does medical training promote or deter self-directed learning? A longitudinal mixed-methods study.

자기주도학습 - 개념과 맥락의 중요성 (Med Educ, 2005)

Self-directed learning – the importance of concepts and contexts

G C Greveson & J A Spencer

두 가지 논문 모두 교육자에게 교육접 접근법과 이니셔티브의 효과를 정확히 평가하려면 거기에 깔린 개념을 정확히 이해할 것을 강조한다. Candy는 SDL을 바라보는 여러 상이한 개념이 있다고 말한다. 다양한 교육철학에 따라서 이념적인 것부터 도구적인 것까지 다양한 세팅에서 다양한 함의를 갖는다. Miflin등에 따르면, SDL에 기반한 graduate medical course를 도입할 때 교사와 학생들이 여기에 대해서 가지는 서로 다른 해석때문에 어려움이 있었다고 말한다.

Both papers highlight that it is crucial that educators understand concepts underpinning educa- tional approaches and initiatives, in order to evaluate whether they are likely to be effective. Candy sug- gested there are several conceptu- ally distinct ways of viewing SDL, based on varying educational phi- losophies, from the ideological to the instrumental, which may have different implications for practice in different settings.3 For example, Miflin et al. reported the difficulties in implementing a graduate med- ical course based on the idea of SDL when there were many differ- ent interpretations of the concept amongst the teachers and students involved.4"

Medical educators are exhorted to adopt SDL with the principal aim of producing learners who can manage their own learning throughout their careers."

그러나 Coffield는 너무 오랫동안 평생학습이 근거도 없이, 연구도 부족하고, 이론도 없는 채로 비판받지도 않아왔다고 지적한다. SDL에 대해서도 마찬가지다.

Yet Coffield caustically claimed that for too long life-long learning has remained an evidence-free zone, under-researched, under-theorised, unencumbered by doubt and unmoved by criticism .5 The same could probably be said of SDL."

Canday와 Schimidt는 한 상황에서 SDL을 강조하는 것이 다른 상황으로도 전이될 수 있는가에 대한 의문을 가져왔다.

Both Candy3 and Schmidt6 ques- tioned whether encouraging SDL processes in one context would enable self-management or learner control to be transferred to other learning contexts."

Norman은 자기주도적 학습자들이 자기평가에도 효과적이어야 하지만, 연구 근거는 많은 보건전문직이 이것을 잘 하지 못한다는 것으로 나타난다.

Norman draws attention to the fact that self-directed learners need to be effective at self-assessment, yet research evidence suggests that many health professionals (pre- or post-qualification) have difficulties with this.7"

Schmidt는 전문가 양성에 있어서 SDL 기술이 지나치게 강조되어왔음을 지적한다.

Schmidt put forward similar arguments for his claimthat the importance of SDL skills in professional practice had been overemphasised.6"

인지주의자들은 학습에 있어서 개인적 특성을 강조하지만 Candy는 자기주도적 학습이란 집단성보다 개인성을 강조하는 형태로 잘못 인도되어 왔는데, 사실 지식과 학습은 본질적으로 학습자를 다른 학습자와의 관계 속에 두는 것이다 라고 주장했다.

Cognitivists stress the private and individual nature of learning. However, Candy claims: The termself-direction has misled many into elevating the individual above the collective – but the nature of knowledge and learning inherently puts learners in relationship with others .3"

Self-directed learning--the importance of concepts and contexts.

의과대학생들은 임상실습에서 어떻게 자기주도적인 방법으로 학습할까? Design-based research (Med Educ, 2005)

How can medical students learn in a self-directed way in the clinical environment? Design-based research

Tim Dornan,1 Judy Hadfield,1 Martin Brown,2 Henny Boshuizen3 & Albert Scherpbier4

영국의 GMC는 의과대학생에게 자신의 학습 방향을 스스로 결정하라고 하면서 의학교육을 새로운 트랙에 올려놓았다. 성인학습자 원리에 따르면 - 이 권고안의 근간이 되는 - 학습자와 그의 열망은 교수-학습 공식에서 중요한 위치를 갖는다. 그러나 '자기주도학습'이란 말은 단일한, 일반적으로 동의되는 의미를 가지고 있지 않는다. GMC는 스스로의 학습요구를 조직하고 관리하는 것을 의미하였으며, 성인학습자 원리를 처음 주장한 Knowles는 SDL을 스스로의 동기를 찾고 교사와 보다 동등한 관계를 갖는 것을 의도하였다.

The UK General Medical Council (GMC) set medical education off on a new track when it called for medical students to direct their own learning.1 Adult learning principles ,2 which underpin that recom- mendation, have served a useful role in putting the learner and his or her individual aspirations back in the teaching)learning equation .3 However, the term self-direction does not have a single, generally agreed meaning. The GMC meant organising and managing one s own learning needs.’1 Knowles, who first articulated adult learning principles, meant finding one’s own motivation and having a more equal relationship with teachers.2"

PBL에서의 자기주도적 행동 성향이 임상교육현장으로 잘 전이되지 않는 것은 (1)낯섦, 복잡성, 권력관계, 정서적 요인 등이 젊은 학습자들로 하여금 임상환경에서 자발적인 행동을 어렵게 하며, (2)외부의 방향제시가 필요하다는 결론에 이르게 한다.

The limited transfer of self-directed behaviour from PBL to placement learning6 led us to hypothesise that: unfamiliarity, complexity, power differentials and 1 emotional factors make it hard for young learners to act autonomously, and more external direction is needed in the clinical 2 environment."

iSUS는 학습 관리 시스템으로서, 교육과정 목표를 알고 있고, 각 학생에 대하여 그 목표에 대한 자기보고를 지속적으로 추적하여 전체 그룹과의 향상정도를 비교하고 관련된 교육을 받을 수 있게 도와준다. 학생은 예컨대 신장요로계 질환을 더 학습해야 한다고 할 때, 신장투석 클리닉을 관련 학습기회의 장소로 찾아주고, 그 장소에 참석할 수 있도록 예약을 해준다. 여기까지 몇 번의 클릭과 7초 정도만 소요된다. iSUS는 1명의 학생에게만 공간이  가능하다는 것을 알고, 다른 학생의 예약을 차단해준다. 이전 학생으로부터의 피드백은 유용한 선택의 길라잡이가 된다. 학생이 다음 번 로그인하면 iSUS는 그 클리닉에서의 경험에 대해 코멘트하도록 권한다.

Intelligent signup system ) iSUS The learning management system, described in detail elsewhere,12 is intelligent in that it knows the curriculum objectives, keeps track of each student’s self-reported progress towards them, compares pro- gress against the whole peer group, suggests signups or other relevant experiences, and helps access them. A student might, for example, see they need to learn more about renal ⁄ urological disease, particularly chronic renal failure, identify the predialysis clinic as a relevant learning opportunity, and reserve a place to attend it. That would take 7 mouse-clicks and a few seconds. iSUS would know there was space for only 1 student to attend, and restrict booking to someone who had no other timetabled activity. Feedback from previous students would be available online to guide the choice. The next time the student logged on, the system would ask for a free text comment about their experience at the clinic."

질적연구방법

Textual materials were coded, back-referenced to the complete transcripts, assembled into an evolving interpretation, and reduced to a single narrative using NVivo software (QSR, Doncaster, Victoria, Australia). Three types of statement were identified: 

• 1 Statements describing the behaviour of a student on a specific occasion in a practice setting, coded as performing , observing , discussing or being taught and subcoded as active ( I did ), passive ( I was directed ) or mutual ( We did ). 
• 2 General descriptions of learning were coded for the degree of autonomy expressed by the student, and sources of guidance or support. 
• 3 Conceptualisations of self-direction. "

한 명의 연구자가 일차 코딩을 수행하고, 다른 연구자들은 자료를 읽고 그 코딩의 bias, 생략된 부분, 의견의 차이 등을 찾아내였다. 최종 결과는 3 가지의 statement의 카테고리를 삼각층량함으로서 도출하였다.

One researcher carried out the primary coding. Three others read the material and sought bias in his coding, omissions, and disconfirmatory statements. The final conclusions were arrived at by triangulation between the 3 categories of statement, representative examples of which are cited."

1. 임상환경에서 학생의 행동유형

1. Behaviour of students in the clinical environment 

1.a. Performing 

1.b. Observing"

1.c. Discussing 

1.d. Being taught 

2. 학습에 대한 일반적 묘사

2. General descriptions of learning 

2.a. Learning autonomously 

2.b. Learning with support 

2.b.1a. Staff behaviours"

2.b.1c. Facilitation"

2.b.1d. Feedback"

2.c. Being told what to learn"

3. 자기주도학습의 개념

3. Conceptualisations of self-direction 

3.a. Definition of self-direction"

3.b. The value of self-direction"

3.c. Prerequisite conditions for self-direction"

3.c.1a. Objectives"

3.c.1b. Student attributes"

3.c.1c. The transition into self-direction"

3.c.1d. Support by teachers"

자기주도학습은 운전석에 앉는 것과 같아 자신의 학습에 책임을 지는 것이다. 종종 그 과목에서 부족했던 부분을 보충하는 것일 수도 있지만, 비효율적인 학습방법으로 인식되기도 한다. 한 학생은 direction (명확한 목표가 있는 것)을 motivation (direction이 선행될 때 가능한 것)과 구분하기도 했다. 자율이란 '어떻게'를 선택하는 것이었으며 '무엇을 학습할 것인가'를 결정하는 것을 의미하지 않았다. 학습의 다양성을 인정하면서도 응답자들은 핵심 목표를 다루고 싶어했다.

Self-direction involved being in the driving seat, or being responsible for one’s own learning. At times, it could be a compensation for deficiencies in the course, but that was seen as an inefficient way of learning (3.a.1). One respondent distinguished direction – having clear objectives – from motiva- tion – for which direction was a prerequisite (3.a.2). Autonomy meant being able to choose how, rather than what, to learn (3.a.3). Whilst valuing diversity of learning, respondents were concerned to cover coreobjectives. "

교육과정 평가에서 처음에는 학생의 태도에 실망했지만, 점차 '자기주도적 태도'에 대한 질문을 갖게 했다. 우리 학생들은 학습의 개념화와 행동적 차원에서 능동적이었지만, 교사가 그들을 지지해줄 때 더 능동적이고 동기부여를 받았다. 다양한 형태의 지원이 가능하다.

Curriculum evaluation has moved us from disap- pointment in students’ behaviour to a more ques- tioning attitude towards self-direction. Our students were active in their actions and conceptualisations of learning. However, they were most active and motivated when teachers supported them. Support took several forms:"

• organisational: opening up learning opportunities, particularly those that involved students in patient care; 
• pedagogic: suggesting objectives or methods, training skills, giving feedback, explaining con- cepts, and 
• affective: giving permission, helping students through the transition to a more independent learning style, nurturing, and placing demands."

본 연구는 문제바탕학습 방법이 왜 임상교육으로 자동적으로 전이되지 않는가를 설명해준다. 방향제시와 동기부여의 원천, 그리고 그들의 관계가 PBL과 임상교육에서 차이가 있었다. 튜터가 관리하는 PBL 그룹에서 불확실성은 그들로 하여금 지적인 호기심을 유발하고 학습목표로 이끈다.

This study confirms and helps explain our previous observation that problem-based methods do not transfer automatically to the clinical environment.6 The sources of direction and motivation, and their interrelationship, are different in PBL and place- ment learning. Uncertainty in a tutored PBL group in a seminar room motivates students by generating epistemic curiosity 14 and leads them to learning objectives."

본 연구와 다른 연구들은 자기주도적 학습이 임상환경에서의 기본적 전문성 교육에는 적용하기 어려움을 보여준다. 또한 Knowles 자신조차도 나중에는 정서적 지지가 중요함을 강조한 바 있다. 다른 연구자들은 자기주도적 행동과 주변 환경과의 상호작용을 강조한다. 이는 교육에 대한 인지적 이해(cognitive perception)에서는 잘 드러나지 않았던 것이다. Miflin 등은 교사가 임상교육의 필수적 조건임을 찾아낸 바 있다. PBL에 cognitive foundation을 제시한 Schmidt조차도 나중에는 임상환경에서 진정으로 자기주도적이 되기 얼마나 어려운지, PBL에서 임상환경으로 전이되는 것이 얼마나 어려운지 강조하기도 했다.

Our own and others’ research leads us to suggest that self-direction , as literally applied by many of our teachers, is inapplicable to basic professional education in the clinical environment. It is note- worthy that Knowles himself re-emphasised the place of affective support in late publications.15 Other writers have emphasised the interaction between self-directed behaviour and contextual factors,16 which are less emphasised in cognitive conceptions of education17 than new social theories.18 Miflin and colleagues found, like us, that the teacher is a vital condition for clinical learning.5 Their analysis anti- cipated, and has strong similarities with, our own critique of self-direction. Schmidt, who gave the problem-based method its cognitive foundation, later reaffirmed its benefits but recognised how difficult it is to develop truly self-directed behaviours and transfer them from PBL to the clinical envi- ronment.19"

우리는 임상교육을 PBL과 상보적일 수는 있어도 서로 별개의 것으로 바라보아야 한다는 결론에 이르렀다. PBL에서는 동료그룹 내에서라도 자율이 핵심이지만, 임상교육에서는 지지적 참여가 핵심 조건이 된다.

We conclude that placement learning should be seen as separate from, although complementary to, PBL. Whereas autonomy (albeit within a peer group) is key to PBL, supported participation is a core condition for placement learning."

적절한 지원이 있으면 의과대학생들은 매우 능동적인 학습자가 될 수 있으며, Harden 등은 이를 직무기반학습 이라는 새로운 관점에서 보았다.

Harden et al. approached the same problem from a different angle when they devised task-based learning .20"

Med Educ. 2005 Apr;39(4):356-64.

How can medical students learn in a self-directed way in the clinical environment? Design-based research.

학부 교육과정이 학생의 SDL에 미치는 영향 (Acad Med, 2003)

Effect of an Undergraduate Medical Curriculum on Students’ Self-Directed Learning 

Bart J. Harvey MD, PhD, Arthur I. Rothman, EdD, and Richard C. Frecker, MD, PhD

1992년, U of Toronto는 전통적인 강의중심의 교육과정을 소그룹의, 문제중심의, SDL을 장려하는 교육과정으로 개편하였다. 개편한 교육과정은 Barrows가 묘사한 "학생이 경험을 통해 자극받고, 그들이 배우는 내용이 미래에 그들이 감당할 책임과 어떻게 연결되는지 깨닫고, 높은 수준의 학습동기를 유지하고, 전문가적 태도의 중요성을 깨닫는" 것을 목표하였다. 우리는 교육과정 변화가 학생의 SDL에 어떻게 영향을 주는지 보고자 했다.

In 1992, at least in part to address this curricular goal and to enhance students’ self-directed learning (SDL), the Uni- versity of Toronto Faculty of Medicine revised its conventional, lecture-based medical curriculum into a “hybrid,” replacing much of the curriculum time devoted to large-group didactic lectures with small-group, problem-based, and SDL opportunities. The resulting curric- ular changes were designed and implemented to achieve the ideals Barrows describes: “That students would be stimulated by the experience, would see the relevance of what they were learning to their future responsibilities, would maintain a high degree of motivation for learning, and would begin to under- stand the importance of responsible pro- fessional attitudes.”4 We undertook this study to begin to learn whether the curricular revision enhanced students’ SDL (and, ultimately, their abilities as effective lifelong learners)."

총 280명의 학생이 참여하였으며, 각각 학년에서 70명 무작위 선택. 이 숫자를 선택한 이유는 60%응답자가 0.80의 study power와 5% difference를 제공하기 때문이다.

Participants A total of 280 students, 70 from each of the four years of the undergraduate med- ical curriculum, were randomly selected fromthe school’s population of 700. We chose this number because calculations showed that 60 respondents per class (85% response rate) would provide a study power of .80 to detect ( .05) a 5% difference among the four years.23"

네 가지 SDL 요소에 따라서 Rayn은 간략한 두 파트로 된 설문지를 개발하여 SDL의 중요도와 자기주도적 학습자로서의 능력을 평가하게끔 했다.

Guided by these four SDL components, Ryan6 developed and administered a brief, two-part questionnaire to assess students’ perceptions concerning the importance of SDL and their abilities as self-directed learners."

두 개의 가장 널리 사용되는 SDL 척도는 SDLRS와 OCLI이다. 타당도와 신뢰도에 대한 여러 연구가 되어있다.

The two most widely recognized, extensively used, and validated instruments for measuring SDL capability and readiness13–15 are Guglielmino’s Self- Directed Learning Readiness Scale (SDLRS)16 and Oddi’s Continuing Learning Inventory (OCLI).17,18 Several assessments of the reliability and valid- ity of the OCLI and SDLRS have been conducted,14 including dissertations re- porting positive associations between instrument scores and SDL activity."

Ryan의 설문지는 SDL의 네 가지 요소에 대한 것이며 그 요소들에 대해서 중요도와 능력을 0부터 6까지 응답하게 되어있다. SDLR은 58개의 명제로 되어 있으며 5점 척도로 응답하게 되어있다. OCLI는 24개의 명ㅈ제로 되어있으며 7점 척도로 응답하게 되어있다.

Ryan’s questionnaire asks respondents to consider the four identified components of SDL and rate each, from low (0) to high (6), on its importance and their ability with the component.6 The SDLRS contains 58 statements (e.g., “I learn several new things on my own each year”) with five-point responses, ranging from “Almost never true of me; I hardly ever feel this way” to “Almost always true of me; there are very few times when I don’t feel this way.” Total scores range from 58 (least ready for SDL) to 290 (most ready).16 The OCLI contains 24 statements (e.g., “I work more effectively if I have freedom to regulate myself”) with seven- point responses, ranging from “Strongly Disagree” to “Strongly Agree.” Total scores range from 24 (least characteris- tic of self-directed learners) to 168 (most characteristic).17,18 Brockett and Hiemstra,14 in their review of the use and validity of the SDLRS and OCLI, concluded that both are well-accepted measures of SDL."

여학생과 남학생은 SDL 점수가 비슷했다. SDLRS와 OCLI Ryan ability 점수는 나이가 높아짐에 따라 높아졌으며, 의예과 이전에 교육 수준이 높을수록 높았다. 그러나 여러 요인을 포함한 MLR 분석에서는 premedical education만이 유의한 요인으로 남았다.

Women and men had similar SDL scores. SDLRS, OCLI, and Ryanability scores increased significantly by age (data not shown) and highest level of premedical education achieved (under- graduate only, masters, or doctoral). However, only premedical education re- mained significant in a multivariate lin- ear regression including both factors."

SDLRS, OCLI, Ryan importance 점수가 학년간 유의하게 달랐지만, 지속적인 경향은 Ryan importance에서만 확인되었다. 

Although a significant between-yeardifference was found for the SDLRS(p .028), OCLI (p .011), andRyanimportance (p .021), a significanttrend by year was only evident forRyanimportance scores (p .007). Thistrend, however, indicated a decrease inperceived SDL importance by curricularyear."

SDL이 학년 간 차이가 컸지만, 교육과정을 진행함에 따라서 지속되는 경향은 없었고 1학년이 가장 높고, 2학년에서 가장 낮은 경향이 있었다.

Although significant interyear SDL differences were found, SDL scores did not follow a trend consistent with progression through the curriculum, with the first and second years having consis- tently the highest and lowest scores, respectively."

우리는 네 개의 SDLRS 문항이 학년에 따라 경향성을 갖는 것을 확인하였다. 그러나 이 모든 문항이 학년이 올라가면서 SDL이 감소하는 것으로 나타났다.

we found four, albeit different, SDLRS items suggesting a trend by curricular year. These, however, all indicated decreasing SDL into senior years."

본 연구 결과가 교육과정이 SDL을 촉진하지 않는다는 것을 보여주나 다른 해석도 봐야 한다.

Although the results of our study suggest that the curriculum does not foster students’ SDL, alternative explanations should be considered."

평가도구들은 SDL 변화를 보여주기에 충분히 민감한가? 두 가지를 보면 그러하다. (1) 모든 세 가지 평가도구가 유사한 결과를 보여준다. 즉, SDL이 premedical education 수준이 높을수록 높아지는 것이다. (2) 세 척도의 유사성이 우연히 나타났을 가능성은 낮다. 또한 응답률이 거의 일정했으므로 응답 편향의 가능성도 낮다.

the instruments sufficiently sensitive to detect SDL progress? Two factors sug- gest that they are: (1) all three instru- ments provide similar findings, with each able to detect the significant in- creasing trend in SDL associated with higher levels of premedical education; and (2) the similarity of results for each of the three measures of SDL ability (i.e., SDLRS, OCLI, and Ryanability) suggests that the significant results ob- served are unlikely to have occurred by chance (i.e., as a result of the multiple comparisons conducted). Further, re- sponse bias is not a likely explanation for the study’s inability to detect a year- by-year SDL trend because the response rates across the four years were uni- formly high—in excess of 85%."

추가적으로 고려할 점은, 이것이 단면연구라는 점이다. 

An additional consideration, how- ever, is the study’s cross-sectional design. Although this design is more efficient than a longitudinal approach, actual changes in SDL are not measured in the same groups of students over time. Instead, the cross-sectional design assumes the comparability of the four classes. Although the admission proce- dures and curriculum were similar for each of the four years, the failure to detect SDL progress over the curriculum could be the result of unmeasured differences between two or more of the classes. 

Effect of an undergraduate medical curriculum on students' self-directed learning.

보건의료인 교육에서 자기주도학습(SDL)의 효과: systematic review (Med Educ, 2010)

The effectiveness of self-directed learning in health professions education: a systematic review

Mohammad H Murad,1,2 Fernando Coto-Yglesias,3 Prathibha Varkey,1 Larry J Prokop4 &

Angela L Murad2

과연 SDL이 효과가 있는가에 대해 답하고자 할 때 어려운 점은 SDL을 정의하는 것의 어려움과 SDL 기반의 교육과정이 매우 이질적이라는 것이다. 1975년 Malcolm Knowles는 가장 흔하게 인용되면서 가장 포괄적인 SDL의 정의를 다음과 같이 내렸다. 

"SDL은 개개인이 이니셔티브를 쥐고, 다른 사람의 도움이 있거나 없는 환경에서 스스로의 학습 요구를 파악하고 목표를 설정하고 학습에 필요한 인적 자원과 물적 자원을 찾아서 적절한 학습 전략에 도입하고, 그 결과를 평가하는 것을 말한다.'

The main challenges to answering this question involve the difficulty of defining SDL and the heterogeneity of SDL-based curricula. In 1975, Malcolm Knowles provided one of the most commonly cited and comprehensive definitions of SDL: ‘SDL is a process in which individuals take the initiative, with or without the help of others, in diagnosing their learning needs, formulating goals, identifying human and material resources for learn- ing, choosing and implementing appropriate learn- ing strategies, and evaluating learning outcomes.’4"

Knowles는 SDL의 여러 주요 요소에 대해서 말했다.

• the educator should be a facilitator of learning and not a content source; 
• learners should be involved in identifying their learning needs, objectives and resources, and 
• learners should be involved in implementing the learning process, should commit to a learning contract and should evaluate the learning process.4

Knowles described several essential components of SDL: the educator should be a facilitator of learning and not a content source; learners should be involved in identifying their learning needs, objectives and resources, and learners should be involved in imple- menting the learning process, should commit to a learning contract and should evaluate the learning process.4"

다양한 교육적 인터벤션은 SDL의 일부 요소들을 공유하며, 아주 제한된 요소만 가지고 그렇게 불리기도 한다.

Numerous educational interventions share some elements of SDL and are often labelled as such,"

그러나 학습자들이 진정으로 자기주도성을 갖추기 위해서는 Knowles가 내린 정의의 다른 요소들도 포함되어야 한다.

Yet, for learners to be truly self-directed, some of the other components contained in Knowles’ definition4 should be incorporated in the learning process"

학습자들은 42%에서는 무작위 배정되었지만, 무작위 배정의 세부 사항이나 질적 평가를 위한 연구의 다른 특징들은 제대로 보고되지 않은 경우가 많다.

Learners were randomly allocated to SDL in 25 studies (42%); however, details about randomisation and other study characteristics needed for quality assessment were poorly reported."

분석한 연구 논문들의 목록

A detailed description of included studies is presented in Table S1."

40개의 연구가 지식 영역, 9개 연구가 술기 영역, 5개 연구가 태도 영역

Forty studies reported outcomes in the knowledge domain, nine studies reported outcomes inthe skills domain, and five reported outcomes intheattitudes domain."

상호작용화된 컴퓨터 모듈 사용이 13개 연구, 나머지는 비-상호작용 (책, 시청각자료) 자료 활용

Learning resources included interactive computer- ised modules in 13 studies (22%) and non-interactive (reading materials, audiovisual resources) in the remaining studies."

지식 영역에서는 중등도의 향상이, 태도와 술기 영역에서는 통계적으로 유의미하지 않은 매우 작은 향상이 있었음.

When data were pooled in meta-analysis and out- comes compared with those of traditional teaching methods, SDL was associated with a moderate increase in the knowledge domain (SMD 0.45, 95% CI 0.23–0.67; I2 = 92%) (Fig. 2), a trivial and non- statistically significant increase in the skills domain (SMD 0.05, 95% CI ) 0.05 to 0.22; I2 = 2%) (Fig. 3), and a non-significant increase in the attitudes domain (SMD 0.39, 95%CI ) 0.03 to 0.81; I2 = 91%) (Fig. 4)."

학습자들의 learning resource를 선택할 경우에 다른 경우에 비해서 지식 영역에 통계적으로 유의미한 향상이 있었다.

There was a statistically significant interaction suggesting that when learners were involved in choosing learning resources, they made larger improvements in the knowledge domain."

SDL 인터벤션의 기간과 effect size 사이에는 상관관계가 없었다. 

In meta-regression, there was no correlation betweenthe observed effect size and the length of SDL intervention (p = 0.64) or the time interval between the completion of the intervention and outcome assessment (p = 0.14)."

SDL은 학습자들이 학습자원 탐색에 참여했을 때 더 효과적이었다.

In addition, SDL seemed to be more effective when learners were involved in identifying their learning resources."

Knowles suggested that learners who are self-directed should consult with educators and determine the methods and resources that best fit their learning style and the curriculum objectives.4 For example, cognitive objectives can be achieved using written resources or panel discussions; behavioural objectives can be attained using role-play and case-based learning, and psychomotor objectives are best fulfilled by role-play and simulation. Simi- larly, self-directed learners should have the ability to choose the learning method that suits their individual learning styles (e.g. a visual learner may choose a video-based method, etc.)."

이렇게 이질적인 분석결과를 다음과 같이 설명할 수 있다. (1)SDL의 잇점은 학습자가 학습 방법, 전략, 자원을 선택할 때 가장 잘 나타난다. (2)초보 학습자보다 advanced 학습자가 SDL로부터 더 많이 배운다 (3)전공(학과)에 따라서 SDL 의 기대 효과가 다르다.

We were able to partially explain the heterogeneity by finding that: (i) the benefit of SDL increases when learners are involved in choosing their learning methods, strategies and resources, a key component that defines SDL according to Know- les;4 (ii) advanced learners may benefit more from SDL compared with less advanced learners, and (iii) learner type (discipline) may also affect the anticipated benefits of SDL (nurses had a larger SMD compared with other health professionals)."

이 리뷰에서 추론할 수 있는 것은 SDL이 기존의 교육방법과 거의 비슷한 수준으로 효과적이라는 것이다. SDL은 특정 상황에서 추천되어왔으며(성인학습자, advanced 학습자, 학교 시설과 교사에 대한 접근이 어려운 상황), 학습할 내용이 많을 때 보조적인 방법으로 사용되어왔다. SDL은 비용-효과적이기도 하다.

Inference from this review implies that SDL is likely to be as effective as traditional learning methods. Self-directed learning has been suggested in certain settings (e.g. for adult learners and advanced learn- ers, and in contexts in which access to academic institutions or teachers is limited) and as a supple- mental method of learning when learning content is large.6 It is also plausible that SDL is cost-effective."

SDL에 대한 다양한 묘사가 존재하지만 Knowles가 아마 가장 포괄적이고 가장 빈번하게 인용될 것이다. 그 정의는 하지만 타당화되지 않았으며, 교육자들은 교육과정을 개발하면서 Knowles가 말한 요소 일부 중 그들의 환경에 활용가능하고 관련성이 있어 보이는 부분만 포함시킨다.

Although several descriptions of SDL exist, that byKnowles4 is perhaps the most comprehensive and most frequently cited. This definition has not beennecessarily validated; thus, educators developing SDL curricula have incorporated some of the elements described by Knowles as they have deemed relevant and feasible in their learning environments"

현재까지는, 우리의 판단은 SDL은 기존의 교육방법에 비해서 세 가지 영역 모두에서 비슷하게 효과적이라는 것이다.

At present and according to our findings, we believe that SDL in health professions education is at least as effective as traditional learning in all three domains."

SDL은 지식 영역에서 더 효과적인 것으로 보인다. 우리는 교육자들이 SDL 교육과정을 도입할 때 다음을 고려해야 한다.

Self-directed learning may preferentially be more effective in the knowledge domain. We recommend that educators embarking on developing SDL curricula for learners in health professions should: 

• involve learners in choosing learning resources and strategies to enable them to find the most appropriate resources to fit their individual learning styles as well as the overall learning objective; 
• consider SDL as an effective strategy for more advanced learners (e.g. those in the later years of medical school or residency and doctors in practice), and 
• consider SDL particularly when the learning outcome falls in the knowledge domain."

Med Educ. 2010 Nov;44(11):1057-68. doi: 10.1111/j.1365-2923.2010.03750.x.

The effectiveness of self-directed learning in health professions education: a systematic review.

"나는 절대 프로 축구선수는 못 될거야" - 자기평가의 오류들 (J Contin Educ Health Prof. 2008)

“I’ll Never Play Professional Football” and Other Fallacies of Self-Assessment

KEVIN W. EVA, PHD; GLENN REGEHR, PHD

자기평가라는 용어가 사용되는 다양한 맥락 각각은 '자기평가'라는 용어의 사용을 정당화해줄지도 모르지만, 종합적으로 보면 그렇게 다양한 개념들을 하나의 이름 아래 두는 것은 교육자와 이론가 모두에게 혼란과 갈등의 원인이 될 뿐이다.

While each of these contexts may, individually, be a justifiable use of the term self-assessment, collectively, couching such very different concepts under a single label can be a significant source of confusion and conflict for educators and theoreticians alike."

Self-assessment라는 용어가 여러 커뮤니티에 수출되어 사용됨에 따라서 모든 것을 포괄하는 용어가 되었고, 결국 "어떻게도 정의되지 않는" 것이 되어버렸다.

It is important, however, to recognize that when such terms are exported to the larger community, they run the risk of becoming so all- encompassing as to include everything and, therefore, ul- timately define nothing."

Lingard와 Haber는 "우리가 쓰는 용어가 우리가 가질 수 있는 사고의 폭을 열어주기도 하고 제한하기도 한다"

Lingard and Haber have stated that “the language we use both makes possible and constrains the thoughts we can have.”4"

Self-Assessment Versus Self-Directed Assessment Seeking"

보건의료인에게 있어 자기평가가 CPD cycle에서 매우 중요하다는 것은 일반적으로 잘 알려진 사실이다. 이는, '자기조절에 능한 전문직'의 원형이 계속적 교육 활동에 대한 길잡이로서 정기적으로 자신의 약점을 찾아내는 것이기 때문이며, 이를 통해서 현실에서의 격차를 좁혀나가는 것이다. 이러한 점에서 '자기평가'란 종종 은연중에 개인적이고, 누군가 지도해주지 않는 와중에 이뤄지는 성찰과정으로 여겨진다. 예컨대 이러한 개념은 Colliver가 말한 "니 점수를 맞춰봐" 형식의 자기평가 연구 모델에 부합하는 것이다. 이러한 연구 결과의 결과는 '자기평가 점수는 대체로 정확하지 못하다'라는 결과를 반복해서 생산해냈다.

It seems generally well accepted in the health professions that self-assessment is a key step in the continuing pro- fessional development cycle. That is, the archetype of the self-regulating professional is seen as one who regularly self- identifies areas of professional weakness for the purposes of guiding continuing education activities that will overcome these gaps in practice.6 In this construction, self-assessment is often ~implicitly or otherwise! conceptualized as a per- sonal, unguided reflection on performance for the purposes of generating an individually derived summary of one’s own level of knowledge, skill, and understanding in a particular area. For example, this conceptualization would appear to be the only reasonable basis for studies that fit into what Colliver has described as the “guess your grade” model of self-assessment research,7 the results of which formthe core foundation for the recurring conclusion that self-assessment is generally poor.8"

이러한 "지도받지 않는, 내적으로 생성되는" 자기평가에 대한 구조는 Boud가 말한 "자기평가란 고립되고 개인적인 활동이 아니라 동료와, 교사와 다른 정보원을 동반하게 된다"와 대비된다.

This “unguided, internally generated” construction of self- assessment stands in stark contrast to the model put forward by Boud, who argued that “the phrase self-assessment should not imply an isolated or individualistic activity; it should com- monly involve peers, teachers, and other sources of in- formation.”9"

Boud가 묘사한 자기평가는 '바깥을 바라보고' '외부적으로 피드백을 찾으며' '외부로부터 정보를 찾고' '이러한 외부 정보원 평가에 활용하여' '수행능력의 향상을 이뤄내는 것'이다. 이러한 측면에서 자기평가는 자신을 평가하는 교육학적 전략 그 이상이며, 한 사람이 마스터해야하는 능력이 아니라, 길러야하는 습관에 가깝다.

The conceptualization of self-assessment as enunciated in Boud’s description would appear to involve a process by which one takes personal responsibility for look- ing outward, explicitly seeking feedback and information from external sources, then using these externally generated sources of assessment data to direct performance improvements. In this construction, self-assessment is more of a pedagogical strategy than an ability to judge for oneself; it is a habit that one needs to acquire and enact rather than an ability that one needs to master."

Self-Assessment Versus Reflection"

인간은 스스로에 대한 총괄평가 결과를 내리는 것을 잘 못한다는 것은 여러 근거로부터 드러난 것 뿐만 아니라, 사람은 원래 이러한 형태의 자기평가를 잘 못하도록 태어난 것이기도 하다. 그 이유로는 여러 인지적 이유(정보 무시, 기억 편향), 사회생물학적 이유(긍정적 전망을 하도록 적응됨), 사회적 이유(동료와 상관으로부터 언제나 적절한 피드백을 받는 것은 아님) 등이 있다.

We, along with many others, have argued ~and continue to believe! that the evidence reveals not merely that humans are poor at producing self-generated summative assessments of their own performance or ability, but that humans are actually predisposed to being poor at this form of self-assessment. There are cognitive reasons ~eg, information neglect and memory biases!,11 sociobiological reasons ~it being adap- tive to maintain an optimistic outlook!,12 and social rea- sons ~eg, not always receiving adequate feedback from peers and supervisors!13"

그러나 사람이 자기평가를 잘 못한다는 결과가 수행능력에 대한 성찰이 무의미한 활동이라는 것을 의미하는 것은 아니다.

The conclusion that humans do not self-assess well, however, should in no way imply that reflection on performance is a useless activity."

즉, 자기성찰은 '왜 환자의 건강상태가 이러한 방식으로 악화되는가'를 이해하는 것, 혹은 '왜 어떤 사회적 관계가 특별히 성공적이었는가'를 이해하는 것 등의 활동이다. 이러한 "왜" 질문에 대한 답을 찾는 것은 교육학적 전략으로 매우 효과적임이 밝혀졌고, 세상에 대한 이해와 스스로 그것을 구성하는 방식에 도움이 된다. 이러한 방식으로 정신적 에너지를 재투자하는 경향의 차이가 진정한 전문가와 경험만 많은 비전문가를 결정짓는 요인이 된다. 그러나 다시 한번 강조하는데, 이러한 형태의 자기성찰적 행동 - 세상을 더 잘이해하기 위한 - 은 자기평가에 대한 능력과 동일하게 평가될 수 없다. '왜' 질문을 효과적인 방식으로 하는 것이 반드시 스스로의 지식과 능력 수준을 아는 것을 필요로 하지는 않기 때문이다.

Thus, reflection involves activities such as try- ing to understand why a patient’s health state deteriorated in the way it did or why a social interaction went partic- ularly well. Exploring these sorts of “why” questions may very well prove to be an effective pedagogical strategy that can lead to better understanding of both the world and the adequacy of one’s own personal constructions of it. Certainly the expertise literature would suggest that the tendency to reinvest mental energy in this way is a defin- ing determinant of who achieves true expert status in any given field and who evolves into an “experienced non- expert.”15 Again, however, promoting those sorts of re- flective behaviors—aimed at understanding the world better—should not be considered the same as promot- ing self-assessment as a mechanism for judging personal competence. Asking “why” in an effective manner does not require insight into one’s own level of knowledge or abilities,"

Self-Assessment Versus Self-Monitoring"

누군가는 자신의 약점을 인지할 수도 있다. 그러나 여전히 그러한 인식이 더 포괄적인 자기개념에 영향을 주지 않을 수도 있다. 부족한 수행능력에 대한 원인을 "~만 아니었으면 되었을텐데"와 같은 방식으로 깎아내리는 것은 매우 자연스러운 현상이다. 그러나 그러한 식으로 우리의 즉각적 수행능력의 효과성에 대해 인식하는 것으로는 그것들이 모여서 정확한 자기평가를 이루는 것을 보장해주지 않는다.

One may recognize weak- nesses in performance, in the moment, but still not have those observations impact upon one’s broader self-concept. It is easy ~and typically automatic! to find reasons to dis- count negative performances by saying things like “If only @insert favourite excuse here# hadn’t occurred.”18 The end result is that a series of moments in which we are aware of the effectiveness of our immediate performance does not guarantee that those moments will be aggregated to generate an accurate self-assessment overall."

한 의사가 자기의 역량을 넘어서는 특정한 상황을 인지하는 능력은 환자 안전의 중요한 결정요인이 된다. 그리고 이것은 그 의사가 지식과 술기의 격차를 효과적으로 극복해나가기 위해서 더 포괄적인 차원에서의 지속적 교육을 받을 수 있느냐보다 중요하다.

A physician’s ability to recognize when a particular situation is beyond his boundaries of compe- tence is likely to be a greater determinant of patient safety than whether or not that physician is able to determine which broad-based continuing education activities would most ef- fectively fill his gaps in knowledge0skill."

자기평가와 자기주도적 평가 탐색을 비교하였다. 또한 자기평가와 자기성찰을 비교하였고, 마지막으로 자기평가와 자기모니터링을 비교하였다.

In this section, we have contrasted self-assessment ~an abil- ity! with self-directed assessment seeking ~a pedagogical strat- egy! as potential mechanisms for determining one’s areas of strength and weakness, we have contrasted self-assessment ~an ability! with reflection ~a pedagogical strategy! as po- tential mechanisms for improving one’s understanding of the world, and we have contrasted self-assessment ~an ability! with self-monitoring ~an immediate contextually relevant re- sponse to environmental stimuli! as potential mechanisms for determining the need to recruit additional resources to facilitate performance in particular situations."

The Illusion of Personal Accuracy in Self-Assessment

"자기평가를 잘 하지 못한다"라는 말을 들으면 가장 먼저 보이는 반응은 "'걔네'는 왜 그렇데?"이다. 한발 더 나아가서 만약에 "우리"가 "그들"로 하여금 자기평가를 잘 하게 만들 수 있을 것이다라는 믿음을 갖기에 이른다. 이러한 반응은 무척 팽배해서 역설적으로 대부분의 사람들이 자신의 자기평가능력이 평균 이상이라고 생각하고 있다.

The most common response to the findings that self-assessment is poor appears to be bewilderment at how“they” can be so bad, with a concomitant belief that if “we” can just get “them” to self-assess as well as we do, then everything will be okay. This reaction is suf- ficiently pervasive that, ironically, the majority of people think they are above average in self-assessment ability.19"

First, judging the quality of physical skills for which there is an objectively observable outcome is probably impor- tantly different from judging cognitive aptitudes or less ob- jective physical skills in that, unlike cognitive aptitudes, the mental processes required to judge the quality of physical performances are different ~often derived from external in- formation conveyed via the senses! fromthe internal mental processes required to enact the performance.20"

두 번째로 여기에는 논리적 헛점이 있다. 자세히 말하자면, 세상을 2x2 테이블로 본다고 하면, 스스로 잘 한다고 여기는 - 스스로 못 한다고 여기는, 그리고 다른 사람이 잘 한다고 인정하는 - 다른 사람이 못 한다고 인정하는 네 가지 조합이 있다.

To elaborate, we might think of the world as a 2 2 table in which there are some activities at which we thrive and others at which we perform poorly, crossed with some activities we think we do well and others we think we do poorly."

마지막으로, 이 특별한 상황은 극단적 상황에서의 추론이며, 흔히 오류를 범하기 쉬운 전략이다.

Finally, this particular example involves a process of rea- soning fromextreme examples, another erroneous rhetorical strategy."

안타깝게도, 이러한 개인 수준의, 오류 투성이의 자기 과신은 교육자로서의 우리가 스스로 하여금 자신의 강점과 약점을 효과적으로 찾아내는 것이 가능하다고 믿게 만든다. 그 결과 우리 교육자들은 해답의 일부가 되기는 커녕 문제의 일부가 되고 있다.

Un- fortunately, it is this personal, flawed self-confidence in our own self-assessment ability that has led us as educators to perpetuate the myth that the effective self-identification of strengths and weaknesses is even possible. As a result, we educators have not only failed to be part of the solution, we have actually been part of the problem."

Truths About Self-Assessment

우리의 뇌는 근본적으로 자신의 능력에 대해서 과도하게 긍정적인 태도를 갖도록 설계되어 있다.

Rather, the tendency to be overly opti- mistic about one’s abilities is a fundamental property of the way our brains are wired.25"

우리의 수행능력이 나쁠 때는 외부 환경을 비난하기가 쉽다.

When we have a poor performance, usually it is easy to find a way to blame external circumstances.18"

그러나 우리는 과도하게 긍정적이라는 일반적 원칙은 확고하다. Gilbert가 쓴 것과 같이 "진화는 이러한 정신적 과정을 우리 동의 없이 뇌에 설치했다는 점에서 MS window award를 받을 만하다."

we are betteroff being overly optimistic is a robust one. As Gilbert haswritten in another context, “Evolution deserves the Micro-soft Windows Award for installing these mental processes inevery one of us without asking permission.”28 "

The problem is not that these mental phenomena exist. It is that people do not appreciate that they are active."

온타리오에서의 PREP는 가정의학 의사들의 역량을 지속적으로 측정했고, '불충분한 역량'의 두 번째 뛰어난 예측인자는 '고립된 직무환경' 임을 밝혔다.

The Physician Review andEnhancement Program in Ontario, charged with evaluatingand assessing family physicians’ ongoing competence withinthe province, have reported that the second best predictor ofincompetence is working in isolation.29"

답은 분명하다. 자기평가라는 것은 절대로 스스로 개발할 수 있는 일반적 기술이 아니다.

The evidence is clear and overwhelming: self-assessment is not and will never be a generic skill that one can develop."

우리는 이러한 부정확한 자기평가가 '우리'의 문제이며 '그들'의 문제가 아님을 다시 강조하고 싶다.

We wish to reemphasize that the inadequacy of self-assessment must be viewed as a “we” problem rather than a “they” problem."

자기성찰 연습는 세상을 더 잘 이해하는 것에 목적이 있다. 

The focus of these exercises is not to determine that one is greator at least good enough, but rather to determine how oneunderstands the world and how one might increase this un-derstanding to the benefit of future performance."

"의사들은 자기평가를 얼마나 잘 하는가?" "어떻게 자기평가능력을 향상시킬 수 있는가?" "어떻게 자기평가 능력을 측정할 수 있는가?"와 같은 연구질문은 폐기되어야 한다. 대체로 연구자들이 자기평가 능력을 향상시켰다고 할 때를 보면 Ward가 묘사한 오류에 흔히 빠져있다. 집단 수준의 상관관계에 지나치게 빠져있거나, 실제 점수와 한 사람이 평가한 자기평가 점수 사이의 차이를 정확도의 척도로 보는 것이다. 대신 우리는 세 가지 분야의 연구를 해야 한다.

We believe that research questions that take the form of “How well do various practitioners self-assess?” “How can we improve self-assessment?” or “How can we measure self-assessment skill?” should be considered de- funct and removed from the research agenda. Usu-ally when researchers claimto have improved self-assessmentthey have fallen prey to one of the fallacies described byWard et al. ~eg, placing undue faith in group-based corre-lations or mistaking distance of individual guesses fromtruescores as a sensible measure of self-assessment accuracy!.30 Instead, we see three potential programs of research that parallel the three concepts we have distinguished from self- assessment in the first section of this article."

이런 것을 염두에 둔다면, 주요 주제는 역량을 어떻게 유지할 것인지, 그리고 어떻게 CPD를 유지할 것인지이다. Schon의 용어를 따르자면 자신의 행동에 대한 성찰은 '자기주도적 평가 탐구'의 습관을 기르고 외부의 피드백 소스를 흡수하여 자신의 장점과 단점을 인지하는 능력을 개발하는 것이다.

Withthis inmind, we wouldargue that the predominant concern regarding the professional issues of maintenance of competence and continuing professional development ~in Schön’s terms, issues of reflection on practice!31 should fo- cus upon developing habits of self-directed assessment seek- ing and upon understanding factors that influence our ability to absorb these external sources of feedback in developing a coherent self-awareness of our strengths and weaknesses."

중요한 것은 한 사람이 그 자신의 점수를 정확히 예상할 수 있느냐가 아니라, 자신의 점수를 알았을 때 무슨 행동을 하느냐이다.

The relevant issue is not whether a person can predict his score on a test, but what he does with the information when he finds out his score."

Self-directed assessment seeking에 대해서 우리가 해야 할 질문은 이런 것이다. 

Thus, we should be asking questions like “What forms of external data would helpindividuals recognize areas that require updating?” “How can we collect and deliver these data in a meaningful form?” “How can we convince people to believe this feedback and incorporate it into their self-concept?” and, more generally, “How can we get people to act on externally derived infor- mation?”"

• “What forms of external data would help individuals recognize areas that require updating?” 
• “How can we collect and deliver these data in a meaningful form?” 
• “How can we convince people to believe this feedback and incorporate it into their self-concept?” and, more generally, 
• “How can we get people to act on externally derived information?”

Self-reflective exercises에 관해서 우리가 물어야 하는 질문은 이런 것이다.

• “Does engaging in self-reflection result in improved performance” could parallel the emerging literature that reveals the pedagogical benefits of externally derived assessment strategies (eg, multiple-choice tests of knowledge).32,33 

• More sophisticated questions could address 
• (a) whether or not sharing one’s self-reflections with peers, a tutor, or a mentor is necessary to elicit full advantage of the activity and 
• (b) whether or not developing the habit of self-reflection in one context tends to transfer readily to maintaining that habit in novel contexts or at variable stages of one’s career

Self-Monitoring에 관해서 우리가 물어야 하는 질문은 이런 것이다. 

Thus, we should be asking questions like “Do individuals show behavioral indications of slowing down0help seeking when they reach the bound- aries of their knowledge0abilities in their moment-to-moment interactions with patients?” “What cues ~external or internal! initiate such slowing down processes?” “Does the initiation of these processes impact upon the appropriateness of the care provided?” and “Howbest can the skills associated with slowing down and help seeking be taught?”"

• “Do individuals show behavioral indications of slowing down/help seeking when they reach the boundaries of their knowledge/abilities in their moment-to-moment interactions with patients?” 
• “What cues (external or internal) initiate such slowing down processes?” 
• “Does the initiation of these processes impact upon the appropriateness of the care provided?” and 
• “How best can the skills associated with slowing down and help seeking be taught?”

J Contin Educ Health Prof. 2008 Winter;28(1):14-9. doi: 10.1002/chp.150.

"I'll never play professional football" and other fallacies of self-assessment.

OSCE 스타일의 시험에서 평가자의 판단이 대조효과에 영향을 받을까? (Acad Med, 2015)

Are Examiners’ Judgments in OSCE-Style Assessments Influenced by Contrast Effects?

Peter Yeates, MClinEd, PhD, Marc Moreau, MD, and Kevin Eva, PhD

불행하게도, 판단-기반 평가는 psychometric weakness를 안고 있다. 그리고 이러한 문제는 reformulation이나 training으로도 충분히 극복하지 못해온 것이 사실이다.

judgment-based assessments are susceptible to a raft of psychometric weaknesses1–3 that have not been satisfactorily resolved through either reformulation4,5 or training.6–8"

평가자의 오류들: "평가자의 인식에 대한 연구에서, 평가자들은 상대적으로 독특하고 개인별로 특유한 수행능력 이론을 가지고 있으며('우수한 수행능력을 구성하는 것이 무엇인가에 대한 개인적 믿음') (공인된 평가기준이 아닌) 자기 자신의 임상능력을 평가틀(frame of reference)로 활용한다. 평가자들은 평가가 자신의 감정이나 (인색하게 보이기 싫음), 특정 피교육자와의 과거 경험에 영향을 받으며, 조직의 문화에 의해서도 영향을 받는다는 것을 인식하고 있다. 더 나아가서 평가자의 판단은 관찰한 것을 넘어선 추론에 의해 영향을 받기도 하는데, 여기에는 피교육자의 문화, 교육, 동기에 대한 지레짐작까지 포함된다. 이런 연구 결과 외에도 평가자들은 행동 준거에 따라서 평가하라는 지침에도 불구하고 피교육자의 수행수준을 다른 피교육자와 비교하는 경향을 보인다.

“assessor cognition” assessors appear to possess relatively unique, idiosyncratic performance theories (personal beliefs about what constitutes good performance)9,10 and may use their own clinical abilities as their frame of reference (rather than recognized assessment standards) when judging the performance of trainees.11,12 Assessors perceive that their judgments are influenced by their own emotions (e.g., not wanting to feel mean), their prior experiences of particular trainees’ performance, and their institutional culture.12 Further, assessors’ judgments appear to be frequently guided by inferences that go beyond their observations, including presumptions about a trainee’s culture, education, or motivation.12,13 Additional to these findings, an exploratory investigation10 suggested that assessors, despite instructions to judge against a behavioral standard, showed a tendency to make judgments by comparing trainees’ performance against other trainees’.

요약하면, 역량에 대한 이해라는 것은 본질적으로 상대적인 것이라 할 수 있다.

in essence, it indicates that their understanding of competence may be inherently comparative (i.e., norm referenced).17

어떤 순서든 학생이 OSCE 서킷에 들어가면 그 학생과 그 전 학생간의 관계는 무작위가 된다. 그 결과 아주 큰 데이터셋을 분석한다면 특정 학생과 그 전학생간의 상관관계는 전혀 없어야 한다. 정적 상관은 assimilation effect를 의미하며, 부적 상관은 contrast effect를 의미한다. 

When students are not entered into an OSCE circuit in any particular order, the relationship between a performance and its predecessor should be random. Consequently, when a large dataset is examined, no relationships should exist between the scores of performances and their predecessors. A positive relationship between successive performances would indicate an assimilation effect, whereas a negative relationship would indicate a contrast effect. The first dataset was drawn from the 2011 United Kingdom Foundation Programme Office (UKFPO) Clinical Assessment. The second dataset was drawn from the 2008 Multiple Mini Interview (MMI) that was used for selection into the University of Alberta Medical School.

'스테이션 서킷'이란 것은 각 특정 스테이션에 들어가는 지원자의 점수들로 구성된다. 스테이션의 난이도나 평가자의 엄격한 정도가 분석에 영향을 주는 것을 방지하기 위해서 모든 점수를 z score로 변환하였다. 수행능력 기반 평가 점수는 총 5288명을 대상으로 수집하였다. 모든 점수의 평균과 중간값은 비슷했다. skewness와 kurtosis는 -1과 1 사이이다.

A “station-circuit,” therefore, comprised the scores for the candidates at an individual station for an individual circuit of the exam. To prevent station difficulty and/or rater stringency from influencing the analyses, we transformed every candidate score into a z score centered around the mean of its station-circuit. Performance-based assessment data were available for 5,288 candidate observations (see Table 1). All scores’ mean and median values were similar, with skewness and kurtosis values between −1 and 1, indicating that data were adequately normal for parametric analysis.

어떤 관계가 나타나더라도 분석 중 생긴 artifact가 아니라는 것을 확실히 하기 위해서 Excel을 활용하여 Monte Carlo Simulation을 수행하였다. 20개의 무작위 숫자를 생성한 뒤 동일한 분석을 하였다.

To ensure that any relationship observed was not an analytic artifact generated from the way in which scores were compared with preceding scores, we used Microsoft Excel (Microsoft Corporation, Redmond, Washington) to run a Monte Carlo simulation. Twenty random numbers were produced and then used to calculate the same “preceding candidate” metrics as described above (n-1, n-2, etc.).

첫 번째 분석은 contrast effect가 나타나는가였다. 분명하게 현재 점수가 그 앞 세 명의 점수의 평균과 관계가 있음이 나타났으며, 어떤 이전 점수와도 관계가 나타났다.

We initially queried whether the previously demonstrated contrast effects15,16 examiners may be susceptible to contrast effects in real-world situations despite the formality of high-stakes exams and despite explicit behavioral guidance. Notably, the observed relationships were stronger when current scores were related to the average of three preceding scores relative to when they were related to any individual preceding performance.

이론적으로, 이는 우리가 기존에 주장한 바를 지지하는데, 보건의료인력 교육에서 준거지향평가에 의지함에도, 고도로 훈련받고 충분한 자원이 제공되는 평가 참여하는 평가자도 역량에 대한 '고정된(fixed)' 감각이 없다

Theoretically, this sustains our prior assertions15,16 that, despite our espoused reliance on criterion-based assessments within health professional education,19 highly trained and well-resourced examiners may still lack any truly fixed sense of competence against which to judge when making assessment decisions.

평가자들은 특정 수험생을 평가할 때 단순히 가장 최근 수험생과 대조하는 것이 아니라 그 전에 접한 수험생들의 수행능력을 통합하여 기준을 정하는 것으로 보인다. 이는 행동을 설명한 문구나, 훈련에도 불구하고 평가자들은 과거의 예제들을 바탕으로 비교 판단을 하기 위한 mental database를 축적한다는 생각을 지지한다. 이런 효과가 시험의 맨 끝까지 나타난다는 점에서 이것은 워밍업 단계의 현상이 아니며, 따라서 한 두개의 초기 스테이션을 제외한다고 사라질 수 있는 것이 아니다. 가장 강한 부적 상관은 특정 학생을 접하기 4, 5, 6번째에 접한 학생의 평균이였고, 이는 평가자에게 초반부에 접한 케이스가 의미를 갖는다는 초기효과(primacy effect)를 보여준다.

This suggests that assessors mentally amalgamate previous performances to produce a performance standard to judge against rather than simply contrasting with the most recent performance. This is consistent with the idea that, despite the availability of behavioral descriptors and training, examiners accumulate a mental database of past exemplars against which they make comparative judgments. The persistence of the effect near the end of the exam indicates that it is not a “warm-up” phenomenon, and so cannot be counteracted by discarding one or two initial stations. That the strongest negative relationships were observed between later candidates and the average of students that preceded them by four, five, and six places suggests a primacy effect in these exemplar comparisons, in that the early cases that examiners see may be particularly meaningful in setting expectations.

과거 연구들은 대조효과가 여기서 나타난 것보다 더 컸는데, 몇 가지 해석이 가능하다. 첫 번째로 실험실 상황(조작된 상황)에서는 참가자들인 지속적으로 강한, 양방향의 조작(manipulation)에 노출되게 된다. AVN4-6이 가장 큰 영향을 줬다는 것이 특히 시사하는 바는 비디오 기반의 예제를 활용하여 모든 평가자에게 초반의 표준화된 비교측정기(comparator)를 만들도록 할 수 있다는 장점이 있다. 

The preceding laboratory studies showed contrast effects that were larger than those observed in this study, explaining up to 24% of observed score variance in mini-CEX scores. A number of explanations are possible. First, in the laboratory context, participants were consistently exposed to a strong, bidirectional manipulation (either seeing very good or very poor performances prior to intermediate performances). That AvN4-6 showed the largest influence has particularly important implications for practical strategies to overcome the biases observed in this line of research. If confirmed, it would suggest potential benefits in using video-based exemplars to create a standardized set of initial exemplar comparators for all examiners.

Acad Med. 2015 Jan 27. [Epub ahead of print]

Are Examiners' Judgments in OSCE-Style Assessments Influenced by Contrast Effects?

커크패트릭 평가의 네 단계를 열어줄 일곱 개의 열쇠(Performance Improvement, 2006)

Seven Keys to Unlock the Four Levels of Evaluation

by Donald L. Kirkpatrick

First Key: Analyze Your Resources

To do this, you must answer the following questions:

• Does your job consist of only one function—evaluating training programs—or does it include other and perhaps more important duties and responsibilities of planning the curriculum and teaching?

• How large a staff do you have for evaluation?

• How much of your budget can you spend on evaluating programs?

• How much help and cooperation can you get from other departments such as Human Resources or sales if you are evaluating sales training programs?

• How much support and help can you get from line managers if you are training their subordinates in programs such as Leadership Development for Supervisors?

Second Key: Involve Your Managers

If you are going to be effective in evaluating programs, you need to have your managers’ encouragement and support

1. Ask for their input in deciding on subject content. George Odiorne, in one of his books, made the following statement: “If you want people to support your decisions, give them a feeling of ownership.”

2. Get your managers to establish an encouraging climate regarding the program.

3. Ask for their helping evaluating the program. Levels 3 (Behavior) and 4 (Results) require this help. You can evaluate Levels 1 (Reaction) and 2 (Learning) without involving managers because you have control over these two levels. But Levels 3 and 4 are typically influenced by factors beyond those within your control

Third Key: Start at Level 1 (Reaction) and Continue Through Levels 2, 3, and 4 as Resources Permit

Some trainers or HPT professionals refer to Level 1 as “happiness ratings” or “smile sheets,” and I agree! That’s exactly what they are. They measure the reaction of the participants to the program. But those trainers also claim that these evaluations are not of much value. I disagree.

In business, industry, and government, there is a slight difference. First, they may not pay for the program, and the existence of the program doesn’t depend on their attendance. But you can be sure that they will be telling somebody—perhaps even their boss—whether they thought the program was worthwhile.

And when the reactions are positive, the chances of learning are improved.

Fourth Key: Evaluate Reaction

Here are the guidelines for evaluating Reaction:

1. Decide what you want to find out—make a list of items to which you want the reaction of the participants (i.e., subject content, leader’s effectiveness, schedule, audiovisual aids, handouts, case studies, facilities, meals, etc.).

2. Design a form that will quantify reaction. The most common form consists of a five point scale: either Excellent, Very Good, Good, Fair, and Poor; or Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree. The objective is to get as much information as possible in the shortest period of time. Participants are not eager to spend time writing the answers to questions.

3. Provide the opportunity for written comments. End your reaction sheet with the question, “What would have improved the program?”

4. Get 100% immediate response. When the participants leave the program, have them put their completed reaction sheet on the back table. Do not tell them to fill out the form and send it back, or do not ask them to email their reactions. If you do either of these, you will not get enough responses to represent the entire group.

5. Be sure you get “honest” answers. Tell participants you want their honest answers and do not ask them to sign the form.

6. Establish an acceptable standard for their combined reaction and tabulate the forms to see if you achieved or exceeded that standard.

Fifth Key: Evaluate Learning

Here are the guidelines for evaluating Learning:

1. Measure before and after knowledge, skills, and attitudes.

2. Use a form the participants can complete for evaluating knowledge and attitude change.

3. Use a performance test for evaluating skills.

4. Get 100% response.

5. For knowledge and attitudes, design a test that measures what you want them to know and the attitudes you want them to have at the end of the program. 

6. A question that usually arises about the pretest and posttest is whether the same form can be used or if “Form A” and “Form B” should be developed. There are too many problems when you try to develop a “Form A” and “Form B” that will cover the same knowledge and attitudes. So use the same form.

Sixth Key: Evaluate Behavior

Here are the guidelines for evaluating Behavior:

1. Measure on a before-and-after basis if practical. If this is not practical, the alternative is to measure after the program and ask, “What changes in behavior have occurred since you attended the program?”

2. Allow time for behavior change to take place. 3 months or 6 months after the program, or maybe never. The best compromise seems to be 3 months after the program.

3. Use a patterned interview or written survey asking the same questions of all respondents. One important question to include is, “Do you plan to change your behavior in the future?”

4. Decide who will be polled. For example, the following options are possible:

• The participants

• The bosses of the participant

• The subordinates of participants

5. Based on the fact that some participants have not changed their behavior but did answer positively the question, “Do you plan to change your behavior in the future?” repeat the research after 3 more months.

Seventh Key: Evaluate Results

Here are the guidelines for evaluating Results:

1. Measure on a before-and-after basis.

2. Allow time for results to develop—perhaps 6 months or a year.

3. Repeat at appropriate times.

4. Use a control group if practical. A “control” group is individuals who did not attend the program. An “experimental” group is the participants. U

Copyright © 2006 International Society for Performance Improvement

평가에 관한 교수개발: 역량중심교육과정의 잃어버린 고리(Acad Med, 2011)

Faculty Development in Assessment: The Missing Link in Competency-Based Medical Education

Eric S. Holmboe, MD, Denham S. Ward, MD, PhD, Richard K. Reznick, MD, Peter J. Katsufrakis, MD, MBA, Karen M. Leslie, MD, Vimla L. Patel, PhD, Donna D. Ray, MD, and Elizabeth A. Nelson, MD

가장 처음 도입된 역량바탕 프레임워크는 CanMEDS로 1990년대 중반 도입되었다. 이후 ACGME는 레지던트와 펠로우에 대한 일반역량을 개발 및 도입하였다.

One of the first competency-based frameworks to be introduced was CanMEDS in the mid-1990s.14 The Accreditation Council for Graduate Medical Education followed with the development and introduction of the general competencies framework for residency and fellowship in 2001.15

CBME는 복잡한 상황과 맥락에 따른 성과에 좌우되기 때문에 피교육자가 다음 단계로 진행하기에 정말 준비가 되었는가를 평가하기 위한 강건한 평가법과 평가절차가 반드시 필요하다. 그 결과, CBME의 도입시기부터 의학교육자들은 평가도구의 성배를 찾아다녔다.

CBME, because it is driven by complex situational and context-dependent outcomes, requires robust assessment and evaluation processes to determine whether a trainee is truly prepared to enter the next stage of his or her career. As a result, since the inception of CBME, medical educators have been seeking the holy grail of evaluation tools.

그러나 다양한 평가법과 도구들도 비판적으로 학생을 관찰하고, 질문하고, 그들의 수행능력을 실제 환자가 있는 현장에서 판단하는 교수를 대체하지는 못했다. 시험이나 표준화환자로 측정된 피교육자의 능력 또는 역량이 실제 근무지 기반 수행능력으로 '전이'됨을 확인하는 것은 반드시 교수의 책임이 된다. 왜냐하면 발달의 궤적에 있어서 CBME는 더 빈번한, 더 적시의, 건설적이고 실제 상황에서의 평가에 의존해야 하며 '대리지표'로서의 총괄평가에 의존해서는 안되기 때문이다.

However, these methods and tools cannot replace the importance of faculty who are enabled to critically observe, question, and judge trainee performance in actual patient care situations.24 Ensuring that a trainee’s capability or competence, as measured by exams and standardized patients, translates, or “transfers,” into actual work-based performance with patients and families is an essential faculty responsibility.25 Because of its emphasis on developmental trajectories, CBME requires more frequent, timely, formative, and authentic assessment and less dependence on “proxy,” summative assessments.10

전문성의 개발, 그리고 고립된 자기평가의 오류에 대한 근거들은 이러한 관점을 지지한다. 예컨대 SP만을 이용하여 피교육자를 평가하는 것은 비용이 비쌀 뿐만 아니라, 더 중요하게는 정기적으로 지속적 피드백을 주지 못한다.

This perspective is supported by evidence from work in the development of expertise and the perils of isolated selfassessment. For example, exclusively using standardized patients to judge whether a trainee was acquiring competence in clinical skills would not only be expensive but, more important, would not provide the learner with regular and ongoing feedback

또한 많은 문헌이 대부분의 의사들이 자신의 강점과 약점을 외부 자료나 피드백 없이는 발견하지 못한다는 것을 보여준다.

Furthermore, a substantial body of literature clearly demonstrates that most physicians cannot determine their own strengths and weaknesses without external data and feedback.28 ***

평가자로서의 교수: 도전과 기회

Faculty as Evaluators: Challenges and Opportunities

분절된 학습환경

The fractured learning environment

현재 의과대학 교수는 주로 임상현장에서 피교육자와 근무하게 되고, 이 말은 외래, 클리닉, 병동, 수술장, ICU 등의 microsystem을 의미한다. 이런 임상현장은 근무지기반 훈련과 평가가 이루어지는 맥락이라 할 수 있다.

At present, medical faculty work with trainees primarily in clinical units, referred to by some as microsystems, such as an ambulatory clinic or officebased setting, a hospital ward, a surgical suite, an intensive care unit, or other such sites.29 These clinical units are the context for work-based training and assessment.

피교육자가 열악한 임상 microsystem에서 교육을 받는다면 임상현장에서의 역량, 질적 향상, 시스템 기반 진료를 충분히 배우고 있는지를 알기가 어렵다.

It is hard to conceive that trainees can effectively acquire competency in clinical care, quality improvement, or systems-based practice if they practice in poorly functioning clinical microsystems.

입원환자에 대해 보면, 너무 많은 교수들이 그들의 교육하고 평가하는 장소에 아주 잠시만 머무를 뿐이다. 예를 들면 내과나 소아과 교수는 입원 환자 회진을 2주~4주 정도만 돌게 된다. 이러한 로테이션 구조는 전문과 수련 문화에 너무 깊이 뿌리내리고 있지만, 이러한 microsystem에서의 로테이션이 학습자의 역량에 대한 평가에 대한 교수의 능력에 어떤 영향을 주는지 잘 알지 못한다.

In the inpatient setting, too many faculty are transients in the very clinical units where they teach and assess. For example, faculty in internal medicine and pediatrics often rotate on inpatient clinical services for just two to four weeks. This rotational structure is deeply ingrained within these specialty training cultures, yet we know little about how rotating through these microsystems affects the faculty’s ability to accurately assess competence of their learners.32

현재 의학교육 시스템에서 환자 경험과 시간의 불연속성은 지속적 평가와 피드백을 어렵게 한다. Hirsh 등은 이러한 '연속성'을 의학교육의 "조직 원리"로 강조해야 한다고 주장한다.

This lack of continuity in both patient experience and time with faculty for trainees in the current medical education system makes longitudinal assessment and feedback very difficult. Hirsh and colleagues34 argued for the importance of continuity as an “organizing principle” for medical education

분절된 학습환경과 연속성의 부족은 교수들로 하여금 자신들이 받은 느낌을 동료교수들에게 전달하는 "feed forward"를 주저하게 만든다. 그들은 이러한 feed forward가 교수들에게 편견을 심어줄까 걱정한다. 그러나 그 결과는 정보의 공유를 통한 피교육자의 발달 및 의미있는 전략 수립이 아니라 '매 번 새로 시작하는' 평가일 뿐이다.

Compounding the fractured learning environment and lack of continuity is the substantial reluctance on the part of faculty to “feed forward” information to their colleagues about trainees over fear of “biasing” the receiving faculty.36,37 However, the end result is a perpetual cycle of “starting over” with assessment instead of using the shared information for the trainee’s development and creation of meaningful action plans.

교수들은 시스템의 과학을 이해해야 하며, 어떻게 학제간 팀에서 효과적으로 일하는가를 알아야 하며, 자율성에 대해서 전통적 과점이 아니라 보다 '관계중심적'관점으로 초점을 옮겨야 한다. '관계적 자율성'에서는 인간은 서로 연결되어있으며 상호의존적이고, 자율성이라는 것은 사회적으로 구성되며 다른 사람에 의해서 주어지는 것임을 의미한다.

To do this, faculty must understand the science of systems and how to work effectively in interdisciplinary teams, and they must move away from traditional views to a more relational view of autonomy. Relational autonomy recognizes that human agents are interconnected and interdependent, meaning that autonomy is socially constructed and must be granted by others.42,43

미래의 교수개발은 어떻게 시스템 요인이 교육과 진료의 질에 영향을 주며, 시스템 기반 진료환경에서 교수들이 피교육자 역량 평가을 위해 어떤 준비가 되어야하는가를 다뤄야 한다.

Future faculty development will need to incorporate training about how system factors affect the quality of both teaching and patient care, and also how faculty must be prepared to assess their trainees’ competencies in systems-based practice

몇 가지 이유로 외래환경이 입원환자보다 지속적 평가와 피드백에 더 적합하다.

For several reasons, the outpatient setting holds potentially more promise than inpatient settings for longitudinal assessment and feedback for most specialties.44 

First, many trainees in specialties such as internal medicine, family medicine, and pediatrics work with a stable group of faculty preceptors who can observe these trainees over time.24 

Second, because trainees often have their own panel of patients, assessment methods such as a medical record audit can be combined with reflection guided by faculty.45 

Finally, as so much of medicine has moved into the outpatient setting, it follows logically that more training and assessment should occur here as well.

전통적 평가 역할 

Traditional assessment roles

예측가능한 미래에 두 가지의 역할이 계속 중요할 것이다. (1)지식과 임상추론의 프로빙을 위한 질문, (2)면담, 신체진찰, 상담, 기타 의사소통 기술에 판단을 위한 직접 관찰

For the foreseeable future, two traditional faculty roles in assessment will continue to be essential: (1) questioning to probe knowledge and clinical reasoning and (2) direct observation to judge the clinical skills of medical interviewing, physical examinations, counseling, and other communication skills as well as procedural skills.

교수들은 추론 과정을 강조하는 질문 기술을 개발해야 한다.

Faculty need to develop the skills to ask questions that emphasize the reasoning process and incorporate key findings and lessons from a growing body of evidence from research on cognition.46,47 Practical approaches exist to help faculty acquire these skills.46,48 These questioning skills apply equally well to the evaluation of procedural skills. ***

비록 교수들이 피교육자의 수행능력을 비판적으로 정확하게 관찰할 수 있어야 하지만, 일부 연구결과는 교수들이 피교육자의 부족한 점을 정확히 찾아내지 못한다는 것을 보여준다.

Although faculty need to be critical and accurate observers of trainee performance, limited published research demonstrates that faculty frequently fail to identify deficiencies in trainees’ clinical skills.24,49–51

확실하게 하기 위해서 교수들은 기본적 psychometric과 quality properties에 대한 평가가 이루어진 도구를 활용해야 하며, 최근의 연구는 최소한의 질적 기준을 만족하는 관찰도구들을 정리했다. 그러나 평가서식을 재설계하는 것이 전체 평가의 10%만을 설명한다는 점에서 의학교육자들은 교수들을 더 효과적인 관찰과 평가로 눈을 돌려야 한다.

To be sure, faculty should only use tools that have been evaluated for basic psychometric and quality properties, and a recent systematic review identified a small group of observation tools that meet minimal quality criteria for use.55 However, given that the redesign of evaluation forms only explains up to 10% of the variance in ratings,56 medical educators must now shift their attention to developing more effective methods to train faculty in observation and assessment.

추가적으로 교수들을 숫자에만 기반한 스케일 평가에만 유지하지 않도록 해야 한다. CBME는 질적 평가에 상당히 의존하게 될 것이다.

In addition, we must help faculty and programs move away from rating scales based on just numbers, as CBME will require a greater reliance on descriptive or “qualitative” assessment.57

교수들은 숫자로 하는 평가는 피교육자에 대한 판단을 종합하여 대표할 수 있게 나타내는 것에 그치지 않는 다는 것을 인식해야 한다. 궁극적으로 평가도구는 그 도구를 사용하는 사람의 수준 만큼만 좋은 것이다.

Faculty need to recognize that numeric ratings are nothing more than a process to synthesize and then represent a composite judgment about a trainee. Ultimately, evaluation tools are only as good as the individuals using them;

Albanese 등은 교육 커뮤니티와 기관이 어떻게 교수개발을 구성해야 하는가에 대한 유용한 프레임워크를 제시한다.

Along those lines, recent work by Albanese and colleagues60 provides a useful framework about how the educational community and institutions might structure faculty development activities using an integrated systems model (ISM).

• Changes in assessment and supervision that are also mission critical for the institution and help to build system

“reserve” will be more likely implemented.

• The further a faculty member moves along the stages of change, the higher the likelihood of adoption that can also produce individuals more likely to become champions for the change. 

• Enlisting the assistance of respected educational faculty to help implement the change helps to promote broader

and more rapid uptake by other faculty.

• Helping faculty mentally picture how the change in the educational program will affect and improve their own educational practices will also assist in the adoption of new knowledge and skills.

Assessment by faculty must be grounded in the principles of CBME

CBME에서는 준거와 발달을 기반으로 한 평가가 필요하다. 발달적 용어로 준거를 정의해야 하는데 이는 흔히 milestone이나 benchmark라고 명명되고, 교수나 프로그램 관리자들로 하여금 피교육자들이 적절한 '궤적'상에 있는가를 판단할 수 있게 한다.

CBME requires assessment be criterion based and developmental. Defining the criteria in developmental terms, commonly called milestones or benchmarks, allows faculty and program directors to determine whether the trainee is on an appropriate “trajectory.”62

Milestone은 교육과정과 평가의 청사진이 될 수 있다.

Milestones, in effect, can become the blueprint for curriculum and assessment.62

다양한 연구로부터 평가의 가장 큰 문제 중 하나는 교수들이 적절한 수행능력이 어느 정도인지에 대한 동의가 부족하다는 것이다. 이와 같은 교수진 내에서의 합의의 부족은 신뢰도와 타당도의 가장 큰 적이다. 또한 피교육자들에게 교수로부터 받는 이질적인 평가와 피드백에 대한 부당한 짐을 지우게 된다.

Multiple studies highlight that one of our biggest and most refractory problems in assessment is the lack of agreement among faculty about what constitutes satisfactory performance across competencies regardless of the competency framework.20,54 This lack of agreement among faculty is a major threat to the reliability and validity of decisions about trainee competence.54,56 In addition, it places an unfair burden on trainees to make sense of the disparate ratings and feedback they receive from faculty.

궁극적으로 교수들은 피교육자를 다른 교수에게 넘길 때 의미있는 수행능력 자료를 제공하는 것에 대한 두려움을 벗어나야 한다. 이는 특히 현재 우리의 로테이션 모델에서 중요하다. forward feeding 없이는 피교육자들은 피상적, 비구체적 평가와 피드백에 머물고 말 것이다.

Ultimately, faculty must become less fearful of providing meaningful performance data—including strengths and developmental needs— about the trainee during educational handoffs.36,37 This is especially important in our current rotational model of training—without “forward feeding” of information, trainees may end up in a perpetual cycle of superficial, nonspecific assessment and feedback..

Assessment requires competent faculty 

교수의 임상역량은 효과적 평가를 위해서 중요한 요소이지만, 아직까지 이 부분은 충분한 관심을 받지 못하고 있다. 여러 프로그램들은 교수들이 - 비록 아주 높은 수준은 아니더라도 - 교육과 평가에 충분한 지식/술기/태도 역량을 갖추었음을 전제로 하고 있다. 그러나 우리는 여러 학생들과 전공의가 임상 술기에 상당히 부족한 부분이 있음을 알고 있고, 따라서 나중에 이들이 교수가 되었을 때 중요한 부분이 부족할 수 있다는 것이 그다지 놀라운 사실만은 아니다. 여러 문헌들이 이를 뒷받침하는데, 심장 청진기술에 대한 연구에서 교수들은 3학년 학생들보다 딱히 더 나은 기술을 가지고 있지 않았다. 다른 연구에서는 가정의학의사, 인턴, 외과의사들이 informed decision making skill이 부족함을 보여줬다.

Clinical competence of faculty is a crucial component of effective assessment, yet this issue has received little attention to date. Programs operate on the assumption that faculty possess sufficient, if not high, levels of knowledge, skills, and attitudes in the competencies they are responsible for teaching and assessing. We have known for some time that numbers of students and residents graduate with significant deficiencies in clinical skills,24 so it might not be surprising that those who later become faculty may possess important deficiencies in clinical skills. A growing body of literature supports this concern. For example, a study of cardiac auscultation skills found that faculty were no more skilled than thirdyear medical students.68*** Another study highlighted substantial deficiencies in informed decision-making skills among family medicine physicians, internists, and surgeons,69***

이러한 결과가 시사하는 바는 CBME를 위한 교수개발은 평가능력에 대해서 임상술기 수련만 필요한 것은 아니라는 점이다. 새로운 21세기에 필요한 역량 개발이 필요하다. 대부분의 교수들은 이러한 역량에 대한 어떠한 공식적 교육도 받은 적이 없다. 그 결과 피교육자가 그러한 역량을 학습할 때 교수도 같이 학습하게 되며, collaborative model이 더욱 필요한 것이다.

The implication of these findings is that CBME-focused faculty development will need to incorporate clinical skills training with training in assessment. In addition to improving the clinical skills of faculty, faculty development will also need to incorporate training in the “new” competencies crucial to 21st century practice: evidence-based practice using point-of-care clinical decision support and information; health information technology; teamwork; care coordination; systems functionalities; advocacy; and contextaware professionalism, to name a few. The majority of faculty working today never received formal training in any of these competencies.29 In effect, there are a number of new competencies that faculty will need to learn as their trainees learn them, necessitating more collaborative models of faculty training.

한 명의 교수가 모든 부분에 전문가가 되어야 한다는 것도 안디ㅏ.

This is not to say that a single faculty member need be an expert in all competencies; rather, trainees should be taught and evaluated by those individuals that truly possess the highest level of knowledge and skill in the domain of interest, and those individuals may not be physicians.

Faculty as coach and mentor in assessment

대부분의 피교육자는 감독자 없이 진료를 하게 된다.

Ultimately, the majority of trainees will graduate from their programs and enter unsupervised practice.

포트폴리오는 스스로에 대한 평가를 할 수 있게 도와주는 강력한 도구이다.

Portfolios are a potentially powerful tool for engaging trainees in their own assessment.71

Next Steps: Preparing Faculty for the CBME Era

교수개발이 CBME의 속도결정단계라는 합의가 늘어나고 있다.

There is a growing consensus that the rate-limiting step in the evolution to CBME is faculty development.72

교수가 교육, 평가, 피드백을 포함하는 코치로서 전문가 역할을 할 수 있어야 한다.

The role of faculty as expert “coaches” must encompass teaching, assessment, and feedback.

아직 우리는 효과적인 교수개발 모델이 없다.

We have yet to develop the most effective faculty development models. The good news from a recent systematic review is that the faculty who participate in educational training activities report 

(1) high levels of satisfaction, 

(2) positive changes in their attitudes, 

(3) increased understanding of educational principles and teaching skills, 

(4) changes in behavior as noted by their students, and 

(5) greater involvement in teaching.75

교수개발에서 행동 변화까지.

However, few studies have investigated whether faculty training translates into actual behavior changes among trainees. In addition, most faculty development is designed as a one-time “bolus” activity and less often as a longitudinal designed program.

Acad Med. 2011 Apr;86(4):460-7. doi: 10.1097/ACM.0b013e31820cb2a7.

Faculty development in assessment: the missing link in competency-based medical education.

바람직한 변화에 도달하는 길: 성과바탕의학교육의 다음 단계에 관한 생각 (Acad Med, 2015)

Achieving the Desired Transformation: Thoughts on Next Steps for Outcomes-Based Medical Education

Eric S. Holmboe, MD, and Paul Batalden, MD

Change and OBME

성과바탕(역량중심, 역량바탕, 등등)의학교육의 개념이 새로운 것은 아님. McGaghie 등은 WHO의 1978년 보고서에서 OBME가 지역인구의 요구에 더 잘 맞는다고 주장함. 약 40년이 지난 지금 OBME는 전세계적으로 확실히 변혁을 이끌고 있음. 변화는 불가피하지만 (헤라클리투스는 '변하지 않는 유일한 것은 변화이다' 라고 했지만) 동시에 매우 어렵다. 왜 변화가 어려운지를 이해해야 한다.

Outcomes-, or competency-based, medical education represents a fundamental shift in educational philosophy and perspective,9–11 yet the idea is not new. McGaghie and colleagues,9 writing on behalf of the World Health Organization in 1978, argued that OBME could better meet the needs of populations in local contexts. Now, nearly 40 years later, OBME is clearly driving substantial change and disruption across the globe. Although change is inevitable— the Greek philosopher Heraclitus (402 BCE) observed that the only constant is change (“you could not step twice into the same river”)12—it is often extremely difficult. Understanding why change can be so hard may help elucidate some of the barriers and challenges to implementing OBME.

첫 번째로 Heifetz와 Linsky의 말처럼 변화는 새로운 대안에 대한 것이 아니라, 기존에 존재했던 것에 대한 익숙함을 상실하는 것에 더 가깝다. 의사들은 이미 플렉스너의 GME의 시간-기반 모델을 위해 많은 희생을 해왔다.

First, we echo Heifetz and Linsky,13 who have argued that change is often not about the new alternative but, rather, about the loss of its familiar antecedent. Many physicians and others have made substantial personal investments and sacrifices to lead and serve in educator roles in Flexnerian and time-based models of GME.

두 번째로 우리의 경험을 떠올려보면 '사실'이라고 여겨왔던 많은 것들이 있다. 현재 전공의 교육의 구조는 이미 그것이 효과적이지 않다는 많은 증거에도 불구하고 '사실'이라고 여겨진다. 결국 실험과 혁신은 매우 최소한으로만 시도된다. 과거의 모든 경험이 나쁘다고 말하는 것이 아니라, 일부 요소들은 분명 역사와 경험이 있지만, 우리가 마주한 도전은 기존의 가정, 현재의 상황을 다시 검토하고 그것들을 모두 통합적, 비판적으로 평가하는 것이다. 이 노력의 일환으로 우리는 "서로 상반되는 두 가지 생각이 긍정적 긴장상태를 유지하도록 하는 능력"을 향상시킬 필요가 있다.

Second, our past experience leads us to accept many assumptions as “truths” when we should be questioning them for what they are: assumptions.15 The current structure and architecture of residency have become “truths” despite the disconfirming evidence that they are not always effective; that change is needed; that, minimally, experimentation and innovation should be allowed and welcomed. We do not mean to imply that all aspects of our past are “bad.” To the contrary, some elements of our educational history and experience, such as bedside rounding, remain instructive and valuable.16–18 Our challenge is to embrace the integrative thinking that allows us to consider assumptions, current reality, and critically assessed practices all together. As part of that effort, we need to increase our ability “to hold two conflicting ideas in constructive tension.”19

마지막으로 OBME를 도입하는 것은 Theory U framework을 따른다. 이것은 왜 새로운 것의 도입이 그렇게 어려운지를 보여주는 것이며, 여기서 U는 그 도입과정을 보여준다. 왼쪽(하향)에서는 새로운 것이 등장할 수 있도록 원래 것이 자리를 열어주는 과정이며, 더 나은 것이 우측에서 나타난다.

Finally, the implementation of OBME and other recent changes to the U.S. accreditation system invite attention to Scharmer’s21 Theory U framework, which illuminates why implementation can be so difficult. The “U” describes the journey to implementation: The left-hand side, or down stroke, represents letting go so that something better can emerge; the something better is represented by the upstroke or right-hand side of the U.

의료시스템에는 단순한 문제, 복합적 문제, 복잡한 문제가 있다.

These health care systems involve simple, complicated, and complex problems.23 

• Simple problems can be solved by following an established recipe. 
• A stable, reliable solution or outcome to complicated problems involves multiple steps and requires critical formulas and expertise from multiple fields. 
• For complex problems, however, formulae and algorithms have limited value; each problem usually requires attention to shared purpose and relationships, often involving unique and iterative approaches.23

Richard Normann은 서비스 로직(service logic)의 중요성을 강조했다. 그는 "생산에서 활용으로, 결과에서 과정으로, 교환에서 관계로 초점을 옮길 것"을 주장했다. "service"라는 단어를 통해서 우리는 환자 돌봄에 거의 가치를 더하지 않는 그런 활동을 의미하거나 수련생에 의해 행해지지 말아야 할 그런 활동을 의미하는 것이 아니다. 슬프게도 GME에서 '서비스'라는 단어는 많은 경우 불필요한 '인턴잡'을 의미하는 용어처럼 되어왔다. 우리는 그것이 아니라 '서비스'를 서로 공유하고 있는 의미있는 직무로 정의내리고자 한다. 과정과 관계에 초점을 두는 Normann에 의해 정의된 서비스로직의 의미를 되새기면서 서비스로직을 새롭게 바라보는 것이 필요하다.

Richard Normann27 highlights the importance of a service logic—that is, “forcing a shift of attention from production to utilization, from product to process, from transaction to relationship.” By “service,” we do not mean clinical care activities that add little to patient care or those activities that should not be performed by trainees. Sadly, in the context of GME, the term “service” has too often become conflated with unnecessary “scut work” performed by trainees. Instead, we define “service” as shared and meaningful work. Recapturing the service logic as described by Normann—focusing on process and relationship—is urgently needed in medical education.

'시간'은 소중한 자원이라기보다는 일종의 '개입'의 한 수단으로 여겨지는 경우가 많다. Ten Cate는 쓸모없는 '잘못된 이분법'을 지적했다. 역량을 키우려면 경험이 필요하고, 경험을 쌓으려면 시간이 필요하나, 시간 그 자체만으로는 역량이 되지 않는다. 시간은 현명하고 생산적으로 쓰여져야 한다.

Time is viewed as an intervention (e.g., a “three-month rotation”) rather than a precious resource. Ten Cate recently highlighted the unhelpful “false dichotomy” surrounding the presumed positive association of greater time and better learning or increased skills.33 Competency requires experience, experience requires time, but time alone does not produce competence. Time should be used wisely and constructively.

우리가 사실로 받아들이는 또 다른 가정은 로테이션, 블록 모델인데 이것을 지지하는 근거는 별로 없다.

Another assumption-as-truth is that the rotational or block model of medical education, used in many medical school clerkships and in disciplines such as internal medicine and pediatrics, is necessary to ensure that learners’ experiences are sufficiently broad and deep. Little evidence exists to support this assumption.

의학교육계는 OBME를 "결과물"을 만들어내는 것으로부터 "서비스"를 강조하는 것으로 새롭게 프레이밍해야한다. "서비스"는 언제나 그것의 수혜자와 함께 '공동생산'되고 '공동창조'된다. 이러한 서비스 수혜자의 적극적 관여를 인정하는 것이 '학습자와 교수자의 공동 작업'이라는 '의료전문성 개발'의 새로운 정의를 만들어내는 길이다. OBME에서의 학습자는 그들의 교육경험과 평가활동을 공동으로 이끄는 자발적 에이전트여야 한다.

The medical education community has to reframe the OBME model from one that produces a “product” or “material good” to one that emphasizes “service.” A “service” is always cocreated and coproduced to some degree by the recipient of the service.24–27,48,49 The recognition of this need for active engagement seems to invite a new definition of “health professional development” as the shared work of teacher and learner. Learners in an OBME system must be active agents coguiding both their curricular experiences and assessment activities.

Wagner 등은 의료에서 '활성화된 환자'의 중요성을 역설했다. 서비스로서의 의학교육도 다를 바가 없다. 의학교육자들은 새롭게 설계된 의료시스템에서 일하고 배울 수 있도록 피교육자를 활성화시켜야 한다. 학습자들은 평가와 피드백을 구하기 위해서 자발적이 되어야 한다.

Wagner and colleagues50 have described the importance of “activated patients” for the development of good care. Medical education-as-service should be no different. Medical education needs activated trainees who are able to work in redesigned care delivery systems and learn through a redesigned curriculum. Learners must also be self-directed in seeking assessment and feedback.51

학습자들에게 그들이 은퇴하기 전까지 사용하기에 충분한 지식을 주입하여 전문직에 안착시키는 것은 불가능하다. 새로운 의사들은 단순한 의학 지식과 술기를 아는 것 이상이 필요하며, '자기평가', '성찰', '진료향상을 위한 지속적 자료 활용' 기술을 가지고 있어야 한다. 이미 지식이 풍요로워졌고, 팀 기반이며, 공동창조하고 공동생산하는 의료계에서 자율적, 독립적 의료진은 더 이상 설 곳이 없으며, 비효율만을 높일 뿐이다.

Attempting to launch learners with enough knowledge into a professional orbit that is set up so that they retire before burning up on reentry is illogical. New physicians need more than just competence in medical knowledge and technical skills; they also require the skills of self-directed assessment, reflection, and using continuous data for improving their own clinical practice.57,58 In a knowledge-rich, team-based, cocreating, coproducing world of health care service, the model of the autonomous, independent practitioner is largely irrelevant and increasingly ineffective.

비록 그것이 전부는 아닐지라도 의학교육자들은 학습과 평가에 새로운 기술을 활용해야 한다.

Medical education must also embrace pedagogical technology alongside personal relationships in learning and assessment. On the technology side, content and some skills can be more efficiently provided through online tools and other forms of simulations (e.g., virtual patient cases; practice tests).

예전에는 개별 의사의 영역이라고 여겨진 진단과 같은 분야에까지도 '공동 절차'가 영역을 넓혀가고 있다. '인지 분산 이론'(distributed cognition theory)와 상황 이론(theory of situativity)은 의료에서 사람들 사이의 상호작용과 맥락이 중요하다는 것을 강조한다. DCT는 한 개인이 환자 관리를 위한 모든 지식을 가질 수 없으며, 따라서 피교육자를 교육하는 것에 대해서도 마찬가지이다. Situativity Theory는 Durning과 Artino가 말한 바와 같이 '지식/사고/학습이 경험에 위치(situated)한다'라는 것을 의미한다. 의학교육은 그 모든 과정에서 group learning, interprofessional, cross-displinary work를 효과적으로 포함시켜야 한다.

Group processes now extend into activities traditionally seen as the province of the individual clinician, such as diagnosis.64 Emerging ideas such as the distributed cognition theory and the theory of situativity highlight the critical importance of human interaction and context in medicine. The distributed cognition theory argues that no single individual holds all of the knowledge necessary to care for a patient or, for that matter, to teach medical trainees,65 and situativity theory, as described by Durning and Artino,66 “refers to theoretical frameworks which argue that knowledge, thinking, and learning are situated (or located) in experience.” The curriculum, at every point along the medical education continuum, needs to more effectively incorporate group learning and interprofessional, cross-disciplinary work.29,65,66

Assessment as Continuous Quality Improvement

지난 세기는 'psychometric' century였다. 완전히 psychometric하고 개개인에 초점을 맞춘 평가법은 21세기 교육과 진료에는 충분하지 않다. 실제 근무지기반평가는 전문가간 팀에 초점을 맞추고, 효과적으로 기술을 활용하고, 의료제공자의 변화하는 역할 초점을 맞춰야 한다.

In addition to a service ethos and an interprofessional focus, we need a more robust national system of faculty development in educational pedagogy, quality, safety, value, and assessment.67 In many ways, given its intense focus on measuring the individual across multiple domains, the last century was the “psychometric” century. A purely psychometric, individually focused approach to assessment is insufficient for 21st-century education and clinical practice.68 The focus on interprofessional teams, appropriate uses of technology, and the changing roles of health care providers, patients, and families translates into an urgent need to improve and expand the role of actual work-based assessments.40,69,70

완벽한 전문가로 전공의를 마치는 사람은 거의 없으나, 대부분의 의사결정은 전문의라면 독립적, 자율적 진료에 준비가 되었다는 믿음을 기반으로 한다. 그러나 GME의 새로운 목표는 "능숙한 평생 공동학습자"라는 개념이 되어야 한다.

Almost no one leaves GME training already an expert, yet most entrustment decisions are based on the belief that an individual who has completed his or her residency is ready to enter independent, autonomous practice. A revised goal for GME might be the codevelopment of a physician ready to enter practice, competent in a particular initial role, and activated to be an effective lifelong learner who is self-directed in seeking ongoing assessment and feedback and capable of continuous reflection on, evaluation of, and modification of his or her emerging roles. Better definition of “a proficient lifelong colearner” might help the medical education community progress toward that goal.

평가는 교육과정에 longitudinally embedded되어야 한다. 이것은 '지속적 습관'이 되어야 한다. 의사들은 그들의 커리어가 '수련'의 특성을 가지는 환경에서 절대 벗어날 수 없음을 인지해야 한다.

Assessment must be longitudinally embedded into the medical curriculum such that incoming evaluation data create continuous feedback loops for learners at every point from medical school to practice. Assessment must become an ongoing “habit”; physicians must recognize that their careers never leave the training environment.

Judah Folkman이 말한 바와 같이 '모든 의료행위는 연습이다'. 우리의 도전은 새롭게 양성할 의사들이 자신의 진료에 대한 자료, 그리고 그 이상의 자료를 활용하여 지속적 학습을 할 수 있게 하는 것이다.

As Judah Folkman76 once noted in his commencement address at the Geisel School of Medicine at Dartmouth, “all medical practice is practice.” Our new challenge is to better prepare future physicians for continual learning, using data both from and beyond their own practices.

Changing Roles and Professional Self-Regulation

정책결정자들과 의료계의 지도자들은 수련을 마친 의사들이 21세기의 의료와 의료요구에 부합하지 않는다고 좌절해왔다.

Policy makers and leaders of health care delivery systems have expressed frustration that physicians graduating from training programs are not meeting 21st-century health and health care system needs— despite decades of certification and accreditation assessment practices and standards. In other words, the graduating medical professional coming into the health care delivery system is not sufficiently prepared or proficient to actually deliver the effective, efficient, timely, safe, equitable, and patient-centered care required.

아마 가장 큰 변화는 총괄평가에서 형성평가, CQI로 옮겨가는 것일 것이다. 이는 규제를 하는 조직과 규제를 받는 조직간의 관계가 변한다는 것을 뜻한다. 유의미하고 중요한 기준을 강화하는 것이 멈추진 않을 것이다. 그러나 새로운 기준과 접근법을 개발하고 도입하는 것은 '공동생산'과 '공동창조'가 필요하다.

Perhaps the greatest change is the shift from almost exclusively summative assessment practices to more formative, continuous quality improvement (CQI) practices. The Next Accreditation System (NAS), which includes elements cocreated by members of both the educational community and regulatory groups (i.e., the ACGME, ABMS), represents an initial attempt to shift to a CQI model.77–80 This shift means that the relationship between the regulated and regulators is also changing. Enforcement of meaningful and important standards will not cease: The public expects and deserves nothing less. However, the development and implementation of new approaches and standards requires coproduction and cocreation.24–27

이 지저분하고 어려운 일에는 모든 이해관계자가 co-configuration으로 포함되어야 한다. 근로와 학습의 co-configuration에 반드시 필요한 것은 사용자의 변화하는 요구에 맞는 서비스를 생산하는 것이다.

This messy and difficult work and learning required of all stakeholders (including regulators) must occur in co-configuration. A critical requirement of co-configured work and learning is the creation of services that adapt to the changing needs of the user.81,82 Learning and work involve

(1) adaptive, “customer-intelligent” product-service combinations, 

(2) continuous relationships of mutual exchange …, 

(3) on-going configuration and customization … over lengthy periods of time, 

(4) active customer involvement and input…, 

(5) multiple collaborative producers that … operate in networks…, and 

(6) mutual learning from interactions between the parties involved.82

Achieving the Desired Transformation: Thoughts on Next Steps for Outcomes-Based Medical Education.

가짜 교수의 의심스러운 수업: 의과대학생들의 생각없는 강의평가 (Med Educ, 2015)

A curious case of the phantom professor: mindless teaching evaluations by medical students

Sebastian Uijtdehaage & Christopher O’Neal

강의평가의 타당성에 대한 의문과 비뚤림에 대한 우려가 심해지고 있다. 또한 진실되지 못하다거나 학생들이 생각없이 평가를 해서 그 타당성을 해한다는 연구도 잇다.

Unfortunately, there is a burgeoning body of research from undergraduate and professional courses at North American campuses that casts serious doubt on the validity of SETs,1 suggests they are rife with biases2,3 and even untruths,4,5 and indicates that students may complete evaluations in a mindless manner that further harms the validity of the process.6

우리는 8주 과목에 가상의 교수자를 집어넣었다. 이 교수자에게는 다른 교수들과는 확실히 구분되는, 성-중립적 이름을 넣고, 일반적 강의제목(폐질환 개론)을 넣었다. 학생들은 과목 종료 후 2주 이내에 이 가상의 교수자를 포함한 모든 교수에 대해서 강의평가를 하게 되었다.

We inserted one fictitious lecturer into the evaluation forms for two 8-week, pre-clinical, classroomstyle courses (for the Year 2 class of 2010 and the Year 1 class of 2011). We gave these ‘lecturers’ gender-ambiguous names (e.g. ‘Pat Turner’, ‘Chris Miller’) that were distinct from existing names, and added generic lecture titles (e.g. ‘Introduction, Lung Disease’) (Table 1). Students were required to submit their anonymous ratings of all lecturers, including the fictitious ones, within 2 weeks after the course using our online evaluation system CoursEval (ConnectEDU, Inc., Boston, MA, USA).

학생들은 '적용불가능함' 이라는 옵션을 선택할 수도 있었다.

Students could choose not to evaluate a lecturer by marking the option ‘Not Applicable’.

그 다음 해에, 우리는 같은 절차를 매력적인 젊은 모델의 사진을 넣어서 다시 시행했다. 실제로 교육에 참여한 교수는 23명에서 52명 사이였다.

The following year, we repeated this process (in the classes of 2011 and 2012), but also included a small portrait (150 9 150 pixels) of an attractive young model who, perhaps regretfully, did not resemble any of our faculty members. The number of actual lecturers in each course ranged from 23 to 52, most of whom were depicted in portraits of similar dimensions in our evaluation system.

심지어 많은 학생들은 가상의 교수자의 교육 수행능력에 대한 코멘트를 남기기도 했다. 3명의 학생은 이 강의를 기억하지는 못하지만 이런 강의가 있었으면 한다라고 기술하였지만, 또 다른 3명의 학생은 '정말 훌륭한 강의였습니다' 와 같이 지어내서 기술하기도 했다.

A handful of students even went so far as to provide comments on the performance of the fictitious lecturers. Although three students explicitly stated that they did not recall the lectures but wished they had (‘I don’t think we had this lecture but it would have been useful!’), three other students confabulated: ‘She provided a great context’; ‘Lectures moved too fast for me’, and ‘More time for her lectures’.

이런 생각없는 평가는 근래의 문제만은 아니다. 1977년의 Reynolds의 연구와도 상통한다. 학생들은 취소되어서 실제로는 진행되지도 않은 영화수업에 대해서 실제 진행된 역사수업보다 높은 점수를 주었다. 

Mindless evaluation is not a modern problem. Our findings echo those described by Reynolds in a landmark 1977 paper.7 Like us, he serendipitously found that a vast majority of undergraduate psychology students rated a movie on sexuality higher than a lecture on the history of psychology, although in fact neither event had taken place (due to cancellations). Where our scenario becomes more problematic than that described by Reynolds7 is in the unique structure of a medical curriculum, in which a multitude of instructors teach in the same course and are evaluated in bulk by students.

Dunegan과 Hrivnak은 강의평가의 세 가지 위험요소를 꼽았다.

Dunegan and Hrivnak6 describe three risk factors that may encourage mindless evaluation practices: 

(i) the cognitively taxing nature of SETs; 

(ii) the lack of perceived impact of SETs on the curriculum, and 

(iii) the degree to which the evaluation task is experienced as just another routine ‘chore’. 

Clearly, all of these risk factors may present themselves in a medical school environment.

• 첫 번째: With regard to the first risk factor, evaluating teachers is a cognitively demanding task when it is done conscientiously weeks after the fact.
• 두 번째: Chen and Hoshower9 use expectancy theory to show that students are less motivated to partake in an activity such as evaluation if they fail to see the likelihood that the activity will lead to a desired outcome (e.g. teacher change).
• 세 번째: Lastly, medical students can certainly be forgiven for finding evaluations to be painfully routine and burdensome. As SETs are part of the mainstay of teaching assessment, medical students fill out evaluations numerous times per year during all 4 years of their training