학부 임상맥락에서 전-시험 학습효과의 모델의 작동(BMC Med Educ, 2012)

A model of the pre-assessment learning effects of assessment is operational in an undergraduate clinical context

Francois J Cilliers1*, Lambert WT Schuwirth2 and Cees PM van der Vleuten3




배경

Background


평가와 학습에 관한 문헌은 많지만, "배움을 위한 평가" 노력에 대한 지원을 제공하는 만족스러운 이론이나 모델은 현재 없다.

While there is much literature relating assessment and learning, there is currently no satisfactory theory or model offering support to the “assessment for learning” endeavour.


평가 전(예: 연구 행동), 평가 중(예: 포트폴리오, 시험 효과) 및 평가 후(예: 피드백) 각각 학습에 영향을 미치는 평가 전, 순수 및 사후 학습 효과 사이에 차이가 있다.

A distinction is drawn between pre-, pure and post-assessment learning effects that respectively impact learning before (e.g., study behaviour), during (e.g., portfolios, testing effect) and after (e.g., feedback) assessment.


평가가 학습에 어떻게 영향을 미치는가를 설명하는 검증된 모델(설명보다는)을 통해 학습을 위한 평가의 설계 및 연구에 도움이 될 수 있다.

A validated model explaining (rather than describing) how assessment influences learning could benefit the design of, and research into, assessment for learning.


종합평가는 학습에 큰 영향을 미친다. 우리는 최근 종합 평가의 사전 평가 학습 효과를 설명하는 모델을 제안했다[11]. (그림 1)

summative assessment strongly influences learning. We recently proposed a model explaining the preassessment learning effects of summative assessment [11]. (Figure 1)


효과는 영향 평가, 대응 평가, 인지 기관 및 대인 관계 요인을 포함하는 메커니즘에 의해 매개된다. 따라서, 학생들은 다가오는 평가 이벤트를 고려할 때, 평가가 자신들에게 영향을 미칠 가능성(긍정적이든 부정적이든)과 그 영향의 크기가 얼마나 될 가능성이 있는지를 고려할 수 있다. 그들은 원하는 결과를 가져올 때 주어진 학습 응답의 효과, 그 학습 응답의 비용 및 원하는 결과 또는 가능한 결과가 그들의 가치와 어떻게 관련되는지를 고려할 수 있다. 특정 결과를 가져올 수 있는 능력에 대한 그들의 인식은 또한 그들의 학습에 영향을 미칠 수 있는데, 이는 강사나 동료 학생들과 같은 재학생들의 의견에 대한 그들의 인식과 그러한 인식에 따르려는 동기 부여가 그러할 것이다.

effects are mediated by a mechanism that involves impact appraisal, response appraisal, perceived agency and interpersonal factors. Thus, when contemplating an upcoming assessment event, students may consider the likelihood that assessment will impact them (positively or negatively) and what the magnitude of that impact is likely to be. They may consider the efficacy of any given learning response in bringing about a desired outcome, the costs of that learning response and how the desired or likely outcome relates to their values. Their perceptions of their ability to bring about a particular outcome may also influence their learning, as may their perceptions of the opinions of referents like lecturers and fellow students and their motivation to comply with those perceptions.


Additional File 1: Additional material_Illustrative quotes. pdf for more extensive information on the model.


양적 접근보다는 질적 접근도 적절해 보였다. 전통적인 양적 수단을 사용하여 검증을 시도하기에는 너무 많은 변수와 관계가 있다. 맥스웰[15,16]은 질적 연구가 인과 과정과 이해에서 맥락과 정신 과정의 필수적인 역할을 고려하는 데 적합하다고 주장한다. 그는 또한 인과관계를 이해하는 서비스에서 실험을 사용하기 위한 전제조건에는 결과에 대한 해석을 가능하게 하는 잘 발달된 이론과 "임시적이고 상황적인 변동성"이 없는 조작 가능하고 상당히 간단한 과정이 포함되어 있다고 주장한다[16]. 모델의 아직 잠정적 특성, 모델의 복잡성, 그리고 모델에서 관계의 시간적 및 상황적 가변성을 고려하여, 우리는 모델 개발에서 이 단계에서 정성적 접근법을 선택했다.

A qualitative rather than a quantitative approach also seemed appropriate. There are too many variables and relationships to attempt validation using traditional quantitative means. Maxwell [15,16] argues that qualitative research is well suited to taking account of the integral role of context and mental processes in causal processes and understanding. He further argues that prerequisites for the use of experiments in service of understanding causality include well-developed theory that allows interpretation of the results and a manipulable, fairly simple process free from “temporal and contextual variability” [16]. Given the as yet tentative nature of the model, its complexity and the temporal and contextual variability of the relationships in the model, we opted for a qualitative approach at this stage in the development of the model.



방법

Methods


맥락

Context


Respondents were South African medical students. Most students at this medical school enrolled directly after secondary school.


자료수집 및 분석

Data collection and analysis


Data was collected by conducting in-depth, unstructured, face-to-face interviews with individual medical students. Respondents determined the venue for and language (Afrikaans or English) in which interviews were conducted. Each interview lasted about 90 min, was audiorecorded and transcribed in full. Each respondent was interviewed once. All interviews were conducted by the same author, an educational adviser fluent in English and Afrikaans, with medical and educational qualifications and training in qualitative interviewing. He worked in the faculty but had little day-to-day interaction with students.


Interviews first explored how respondents learned and how they had been assessed across the course of their studies and then how they adapted their learning in response to assessment. The influence of assessment on learning in a theory teaching context was explored first, then that in a clinical teaching context. Interviews were not structured beyond addressing these three issues in these two contexts. Furthermore, in keeping with the grounded theory design of the study, later interviews were informed and influenced by preliminary data analysis of earlier interviews.


Respondents were asked throughout to provide concrete examples to illustrate how assessment influenced their learning. The influence of assessment on learning was thus explored in various and different contexts and across time, although data was collected at one time point. The dataset comprised over 700 pages of transcriptions.


Previously, an inductive analysis of the dataset was undertaken, utilizing the principles of grounded theory(for details, see [11,17]). This initial analysis drew only on those portions of the interviews relating to assessment in theoretical modules and yielded the model described earlier (Figure 1).


Thirty-two students volunteered for interviews.Interviews were conducted with the first 18 students who volunteered (Table 1). The remaining volunteers were thanked but not interviewed. This decision was taken when no new data emerged after interview 14,





결과

Results


결과 개요

An overview of findings


평가 요소와 학습 효과

Assessment factors and learning effects


The relationships between assessment factors and learning effects are summarized in Table 2



기전

Mechanism




모델에 새롭게 등장한 하위요소

A new subcomponent illustrates the operation of the model


The type of model component will be indicated as follows: 

  • 평가요인 AF = assessment factor; 

  • 기전요인 MF = mechanism factor; 

  • 학습효과 LE = learning effect.


규칙적 책무성

Regular accountability


응답자들은 환자에 대한 정보를 제공하고 환자에 대한 질문에 답해야 하는 결과(MF: 영향 가능성)가 지속적으로 임박했기 때문에(AF: Imminence) 임상 실습 모듈의 시작부터 매일의 노력을 학습(LE: 노력의 분배)에 할당하였다.

Respondents allocated daily effort to learning (LE: allocation of effort) from the start(LE:distribution of effort) of a clinical practice module because consequences (MF: impact likelihood) were constantly imminent (AF: imminence), as they presented information on and had to answer questions about patients.


응답자들의 학습은 더 멀리 떨어진 학업적 결과(예: 마크나 진행에 미치는 영향 또는 우수한 임상 치료를 제공할 수 있는 궁극적인 능력)보다 병동 라운드에 정기적으로 노출되는 동안 발생하는 즉각적이고 구체적인 결과(AF: 임미넌스, MF: 영향 가능성)에 더 많은 영향을 받았다. 이는 주기적인 "빈지-러닝"을 유도하는 이론 모듈보다, 정기적이고 균일하게 분포된 "스낵-러닝"(LE: 노력 분포, LE: 노력의 양)을 유도했다.

Respondents’ learning was influenced more by the immediate and concrete consequences (AF: imminence; MF: impact likelihood) that accrued during regular exposure on ward rounds (AF: pattern of scheduling) than by more distant academic consequences e.g., impact on marks or progression, or ultimate ability to deliver good clinical care. This elicited more regular and evenly distributed “snack-learning” (LE: distribution of effort; LE: quantity of effort) by respondents than did the more periodic assessments on theoretical modules that induced periodic “binge-learning”.



자기에게 미치는 영향

Personal consequences


병동 회진에서 질문을 받았을 때 만족스러운 답을 제공하지 못할 경우 심각한 부정적인 개인적 결과를 초래할 수 있다는 점을 고려할 때, 충격 크기(MF)의 평가도 응답자의 학습에 영향을 미쳤다(요약 7 참조).

Appraisal of impact magnitude (MF) also influenced respondents’ learning, given that not being able to provide satisfactory inputs when asked on ward rounds could result in profoundly negative personal consequences (see also Quote 7).


임상적 맥락에서 응답 효과성의 평가(MF)는 종종 평가-관련 성과 목표(즉 접근 목표)보다는, 자신에 대한 부정적인 결과(즉, 회피 목표)를 회피함으로써 자기 목표를 해결하는 것과 더 관련이 있다. 이것은 "내가 이렇게 하면, 내일 나를 '안전하게' 해줄까"의 경우보다 "내가 이렇게 하면, 내가 합격하는 데 도움이 될까"의 경우였다. 부담은 평가와 관련되어있기보다는 개인적인 것이었지만, 그럼에도 불구하고 고부담이었다.

Appraisal of response efficacy (MF) in the clinical context often related more to addressing self goals e.g., preserving esteem, through avoidance of negative intrapersonal consequences (i.e., avoidance goals) than to addressing assessment-related performance goals (i.e., approach goals). It was more a case of “If I do this, will it keep me ‘safe’ on the ward round tomorrow” than of “If I do this, will it help me pass (well)”. The stakes are personal rather than assessment related, but nonetheless high.


개인적인 결과personal consequence가 두드러지게 나타난 것은 다른 영향력의 원천이 부족해서가 아니었다. 학생들은 모듈 및 학년을 통과하기 위해 연속적이고 졸업 평가를 위해 만족스러운 점수를 얻어야 했다.

The prominence of personal consequences was not due to the lack of other sources of influence. Students had to achieve satisfactory marks for continuous and for exit assessment to pass the module and the year.


감정

Emotional valence


학습 환경의 정서적 밸런스(valence)는 개인적 결과와 밀접하게 연관되어 있었으나, 조금은 구별되어 있었고, preceptor들에 의해 발생되었다. 두 가지 극단은 학생들이 여기서 "폭군"와 "테디베어"라고 preceptor를 특징짓는 것에서 구별될 수 있었다.

The emotional valence of the learning environment was closely linked to, but distinct from, personal consequences and was generated by preceptors. Two extremes were discernable from characterizations by respondents of preceptors, called here “tyrants” and “teddy bears”.


폭군으로 특징지어지는 preceptor를 만나면, 배움은 자기 방어적 메커니즘이었으며, 성과 회피 목표 지향에 의해 추진되었다. 충격 가능성(MF)과 충격 크기(MF)의 평가 결과, 병동 라운드에 대한 질책을 피하기 위한 노력(LE: 노력의 할당)이 할당되었다(참고문 4 참조). 응답효과의 평가(MF)는 응답자가 "모든 것"(즉, 해당 병동 라운드의 "안전"에 필요한 모든 것)을 알고 병동 라운드로 와야 한다는 결론을 내리도록 유도하여, 학습에 여분의 노력(LE: 노력의 양)을 할당하였다. 

For preceptors characterized as tyrants, learning was a self-defence mechanism, driven by a performance avoid goal orientation. Appraisal of impact likelihood (MF) and impact magnitude (MF) resulted in effort being allocated (LE: allocation of effort) with a view to avoiding censure on ward rounds (see also Quote 4). Appraisal of response efficacy (MF) led respondents to conclude that they had to come to a ward round knowing “everything” (i.e., everything necessary to “stay safe” on that ward round), so they allocated extra effort (LE: quantity of effort) to learning. 


학생들은 인계student grapevine에 따라서(AF) preceptor가 듣고자 하는 내용을 기준으로 학습할 내용(LE: 콘텐츠 선택)을 선택했다. 컨설턴트의 분노를 피하는 것(MF: 영향 크기)은 좋은 임상의사가 되는 장기 목표와 무관하다고 인식되는 자료조차 학습하는 데 단기 효용(MF: 대응 가치)을 부여했다. 학습자료는 정신 모델(LE: 인지 처리 활동의 성질)에 편입되지 않고 단기 기억력에 전념한 다음, 병동 라운드가 끝나면 민첩하게 잊혀졌다. 

They selected content to learn (LE: choice of content) based on what the student grapevine (AF) indicated was what the preceptor wanted to hear. Avoiding the wrath of the consultant (MF: impact magnitude) gave short-term utility (MF: response value) to learning even material that was perceived as irrelevant to respondents’ longer-term goals of becoming good generalist clinicians. Material was committed to short-term memory, rather than being incorporated into mental models (LE: nature of cognitive processing activities), and then forgotten with alacrity once the ward round was over.


게다가 일단 병동 라운드가 끝나면 학습 의욕도 소멸되고, 더 이상의 보강(LE: 노력의 분배; 감시 & 조정)도 일어나지 않았다. 응답자들은 그들이 이미 알아야 할 것을 알지 못한다는 이유로 조롱당하거나 공격당할까봐 질문을 하지 않을 것이다.

 Furthermore, once the ward round was over, the motivation to learn dissipated, and no further reinforcement (LE: distribution of effort; monitoring & adjustment) took place. Respondents would not ask questions for fear of being ridiculed or attacked (MF: impact magnitude) for not knowing something they should already know.



테디베어로 특징지어지는 프리셉터의 경우, 응답가치의 평가(MF)가 더욱 두드러지게 특징지어졌다. 노력은 "당신의 일을 하는 것doing your bit"으로 보이도록 할당되었다. 학생들은 일반적으로 관련되었다고 볼 수 있는 것, 그리고 관심있는 자료를 학습할 가능성이 더 높았다(LE: 콘텐츠 선택).

For preceptors characterized as teddy bears, appraisal of response value (MF) featured more prominently. Effort was allocated (LE) so as to be seen to be “doing your bit”. Respondents were more likely to learn material of general relevance and of interest to themselves (LE: choice of content).


학습은 mastery approach 지향에 의해 추진되었으며, 자기조절적 강화는 병동 라운드 이후에 이루어진다(LE: 노력의 분배; 조정). 게다가, 감시와 응답자들은 테디베어를 자원으로 인식했다.

Learning was driven by a mastery approach goal orientation, with self-regulated reinforcement of learning taking place after the ward round (LE: distribution of effort; adjustment). Furthermore, monitoring & respondents perceived teddy bears to be resources.


그들은 아무것도 모른다면 병동 라운드의 테디베어에게 물어보고 배울 수 있을 것이라고 느꼈다. 그렇다고 해서 테디베어가 큰 기대를 하지 않았다는 뜻은 아니다. 그들은 그렇게 했다 (제6항 참조). 그러나 그들이 만든 학습 환경은 매우 달랐다.

They felt that if they did not know something, they could ask teddy bears on ward rounds (LE: choice of resources) and learn from them. This is not to say that teddy bears did not have high expectations. They did (see Quote 6). The learning environment they created was very different, however.



고찰

Discussion


우리는 또한 이러한 결과가 실험 조작이 없는 응답자들의 생생한 경험에서 도출된 생태학적 타당성[19]을 가지고 있다고 믿는다.

We also believe that these results have ecological validity [19], being derived from the lived experiences of respondents with no experimental manipulation.


우리의 결과는 모델이 엄격한 총괄평가 맥락의 바깥에서 작동한다는 것을 추가로 증명한다. 총괄 평가는 여기서 학습을 추진하는 핵심 구성 요소는 아니지만, 이 모델은 현재 일어나고 있는 상황에 대한 설명을 제공한다. 평가에 대한 성과에 따르는 결과consequence가 일차적인 고려사항은 아니었지만, 이곳의 조건은 분명히 "결과적consequential"이었다[6]. 따라서 이 모델은 결과적 평가consequential assessment의 사전 평가 학습 효과 중 하나로 더 잘 개념화된다.

Our results further demonstrate that the model is operational outside of a strictly summative assessment context. Summative assessment is not the key construct driving learning here, yet the model offers an explanation of what is happening. Conditions here were clearly “consequential” [6] even though consequences for performance on assessment were not the primary consideration. This model is thus perhaps better conceptualized as one of the preassessment learning effects of consequential assessment.


Preceptor의 역할은 뜻밖의 발견이었다. 슬프게도, 우리의 응답자들이 "폭군" 지도자를 특징짓는 것은 다른 유사한 발견들을 뒷받침한다. 지도자와의 불쾌한 상호 작용은 [20] 스타일이나 교육자 전문성[21]의 문제로 다양하게 특징지어졌으며 무례한 상호 작용, 경시, 굴욕, 괴롭힘, 학대를 수반한다[22-27].

The role of preceptors was an unexpected finding. Sadly, our respondents’ characterization of “tyrant” preceptors supports other similar findings. Adverse interactions with preceptors have variously been characterized as an issue of style [20] or educator professionalism[21] and as involving disrespectful interactions, belittlement, humiliation, bullying, harassment and abuse [22-27].


우리의 결과는 "배움을 위한 평가"의 설계에 대한 실무자들을 위한 지침을 제공한다. [높은 기대]와 [제한된 개인적 결과]로 특징지어지는 [지원적이고 낮은 위험의 학습 환경]은 더 깊은 인지 처리 활동을 낳았다. 이와는 대조적으로, [높은 기대]와 [부정적인 개인적 결과]로 특징지어지는 [부정적, 고위험 환경]은 피상적 인지 처리 전략과 저관여(예: 질문 없음)를 초래했다. 학습에 영향을 미치기 위해서는 평가가 결과적이어야 하며 추상적이고 사소한 것이 아니라 구체적이고 실질적인 결과를 낳아야 한다. 평가를 총괄평가화 하는 것은 평가를 결과적인consequential 것으로 만드는 한 방법이다. 그러나 예를 들어 개인적 부담(존중 또는 주체성)에 영향을 주는 것 또는 감정에 영향을 주는 것은 또한 (긍정적이고 부정적인) 학습 반응을 이끌어낼 수 있다. 즉각적인 결과도 역시 중요한데, 즉각적인 결과consequence가 지연된 결과보다 더 큰 영향을 미치기 때문이다.

Our results yield some guidance for practitioners about the design of “assessment for learning”. A supportive, lowrisk learning environment characterized by high expectations and limited personal consequences resulted in deeper cognitive processing activities. In contrast, a negative, high-risk environment characterized by high expectations and negative personal consequences resulted in surface cognitive processing strategies and low engagement e.g., no question asking. To influence learning, assessment should be consequential and yield consequences that are concrete and substantial rather than abstract and trivial. Making assessment summative is one way of making assessment consequential. However, influencing personal stakes e.g., for esteem or agency and influencing emotion can also elicit (positive and negative) learning responses. Imminence of consequences is also important, with immediate consequences having a stronger influence than more deferred consequences.


이 연구가 제기하는 한 가지 의문점은 어떻게 하면 학생들을 끊임없는 요구로 소진시키지 않고, 평가를 통해 학습 노력의 균등한 분배를 유도할 수 있는가 하는 것이다. 응답자들이 이론 모듈[11]의 평가에 어떻게 반응했는지를 고려할 때, 임상 실습 모듈의 [정기적인 책임감]이 학생들의 전반적인 학습 패턴을 결정하는 데 중심적인 역할을 했고, 후속 이론 모듈에서의 학습에 부정적인 영향을 미친 것이 분명하다. 이론 모듈에서, 응답자들은 모듈의 초반에는 놀다가, 후반부에 목숨을 걸고 "따라잡는다"고 설명했다. 임상 실습 모듈에서는 학습이 그렇게 열외로 취급될 길이 없었다.

One question this work raises is how to induce even distribution of learning effort with assessment without burning students out with unrelenting demands. Considered together with how respondents reacted to assessment in theoretical modules [11], it seems evident that regular accountability in clinical practice modules played a central role in determining students’ overall learning pattern and had a negative knock-on effect on learning in subsequent theoretical modules. In theoretical modules, respondents described taking time off at the start of a module to “catch up” with the rest of their lives. In clinical practice modules, there was no indication that learning could be side-lined like that.


인터뷰에서는 언급되지는 않았지만, 임상 세팅에서의 요구가 그렇게 광범위하게 분산되어있고, 연속적이있었기 때문에, 응답자가 비학업적 측면에 시간을 할애할 기회가 거의 없었을거라 추측할 수 있다. 그 결과 학생들은 임상 세팅이 끝나고 이론 모듈을 시작했을 때 지쳤을 것이며, 잠시 쉬어도 괜찮은 타이밍이 되자, 학생들은 공부를 쉬게 되었다. 물론 그렇게 하는 데 드는 '비용'은 임상 실습 모듈보다 이론적으로 덜 즉각적이었으며, likelihood of impact도 마찬가지였다. 그러나 응답자들은 이론 모듈에서 시험을 준비하는 과정에서 결국 더 높은 학습량의 형태의 '시간차 영향' 있다는 것을 깨달았다그러나 이러한 "시간차deferred" 비용조차 학생들에게는, 그 시간 동안 그들이 삶의 다른 측면에 시간과 관심을 쏟을 수 있는 기회가 주어졌기 때문에 감당할 만한 것으로 간주되었다.

Although not vocalized in interviews, it is tempting to speculate that the demands of the clinical setting were so pervasive and continuous that respondents had little opportunity to take time off to spend on non-academic aspects of their lives; that they were exhausted when they started the next theoretical module, and, being able to do so, took time off from their studies. The costs of doing so were less immediate in theoretical than in clinical practice modules, as was the likelihood of impact. Respondents did realize, though, that there was a deferred impact in the form of a higher workload in the run-up to assessment in theoretical modules. This “deferred” cost was deemed worthwhile, however, for the opportunity to devote time and attention to other aspects of their lives.



Conclusion








 2012 Mar 16;12:9. doi: 10.1186/1472-6920-12-9.

model of the pre-assessment learning effects of assessment is operational in an undergraduateclinical context.

Author information

1
Centre for Teaching and Learning, Stellenbosch University, Matieland 7602, South Africa. fjc@sun.ac.za

Abstract

BACKGROUND:

No validated model exists to explain the learning effects of assessment, a problem when designing and researching assessment for learning. We recently developed a model explaining the pre-assessment learning effects of summative assessment in a theory teaching context. The challenge now is to validate this model. The purpose of this study was to explore whether the model was operational in a clinical context as a first step in this process.

METHODS:

Given the complexity of the model, we adopted a qualitative approach. Data from in-depth interviews with eighteen medical students were subject to content analysis. We utilised a code book developed previously using grounded theory. During analysis, we remained alert to data that might not conform to the coding framework and open to the possibility of deploying inductive coding. Ethical clearance and informed consent were obtained.

RESULTS:

The three components of the model i.e., assessment factors, mechanism factors and learning effects were all evident in the clinical context. Associations between these components could all be explained by the model. Interaction with preceptors was identified as a new subcomponent of assessment factors. The model could explain the interrelationships of the three facets of this subcomponent i.e., regular accountability, personal consequences and emotional valence of the learning environment, with previously described components of the model.

CONCLUSIONS:

The model could be utilized to analyse and explain observations in an assessment context different to that from which it was derived. In the clinical setting, the (negative) influence of preceptors on student learning was particularly prominent. In this setting, learningeffects resulted not only from the high-stakes nature of summative assessment but also from personal stakes, e.g. for esteem and agency. The results suggest that to influence student learning, consequences should accrue from assessment that are immediate, concrete and substantial. The model could have utility as a planning or diagnostic tool in practice and research settings.

PMID:
 
22420839
 
PMCID:
 
PMC3375194
 
DOI:
 
10.1186/1472-6920-12-9
[Indexed for MEDLINE] 
Free PMC Article


+ Recent posts