총괄평가의 맥락에서 피드백의 수용과 활용 장벽(Adv in Health Sci Educ, 2014)

총괄평가의 맥락에서 피드백의 수용과 활용 장벽(Adv in Health Sci Educ, 2014)

Barriers to the uptake and use of feedback in the context of summative assessment

Christopher J. Harrison • Karen D. Ko¨nings • Lambert Schuwirth • Valerie Wass • Cees van der Vleuten

도입

Introduction

모든 형태의 평가 후에 의견을 제공 할 것에 대한 요구가 있어왔다 (Norcini et al. 2011). 그러나 형성평가와 총괄평가 사이에는 인공적인 이분법이 남아있는 경우가 많습니다. 이것은 종종 선생님이 피드백을 제공하고자하는 노력을 제한하지만, 학생들에게는 (특히 이미 통과한 평가의) 피드백에 대한 uptake를 제한합니다 (Archer 2010). 고부담 평가는 학습자에 대한 많은 유용한 정보를 생성 할 수있는 잠재력을 가지고 있지만, 궁극적으로 단일 P/F 결정으로 축소되는 경우가 많습니다.

there have been calls to provide feedback after all forms of assessment (Norcini et al. 2011). However, there often remains an artificial dichotomy between formative and summative assessments. This often limits the effort teachers are willing to put into providing feedback but also limits the uptake of feedback by students especially those that have passed the assessment (Archer 2010). Though high-stakes assessments have the potential to generate much useful information about the learner, they are often ultimately reduced to a single pass–fail decision.

이러한 환원 주의적 접근법에 대한 접근은 문제가 된다(Schuwirth and van der Vleuten 2011). 중요성이 높은 객관적 구조 임상 시험 (OSCE) 평가에서 각 항목이나 스테이션의 점수는 일반적으로 하나의 최종 표시를 만들기 위해 함께 추가됩니다. 

• 의사 소통 기술 OSCE 스테이션에서 만점을 얻었지만 인공 호흡 스테이션에서 0 점을 받은 학생은 

• 의사 소통 기술에서 0 점을 얻었지만 인공 호흡에서 만점을 얻은 점수와 동일한 점수를 받게됩니다. 

이 두 학생의 능력은 크게 다를 수 있습니다.

This reductionist approach to assessment has been challenged (Schuwirth and van der Vleuten 2011). In a high-stakes objective structured clinical examination (OSCE) assessment, the scores for each item or station are typically added together to make one final mark. Thus, a student who scores full marks on a communication skills OSCE station but zero marks on a resuscitation station will receive the same overall score as one who scores zero marks in communication skills but full marks in resuscitation, even though the information learnt about each learner’s competence may be vastly different.

이 인공적인 이분법에 문제를 제기하고, 고부담 평가가 후에 피드백을 제공하려는 시도가 반드시 성공적인 것은 아니다. 제한된 근거로부터 미뤄보면, 학생들은 총괄 written 평가 또는 OSCE 후에 따라오는 피드백을 항상 받아들이지 않거나 제한적으로만 사용한다. 퍼포먼스는 그 한 원인으로 보이는데, 상위권과 하위권 학생들은 피드백을 더 많이 사용하는 경향이있는 반면, 적당히 통과한 (중위권) 학생이 피드백을 가장 적게 활용하는 것으로 나타납니다 (Harrison 외. 2013).

Attempts to challenge this artificial dichotomy and instead provide feedback after highstakes assessments are not necessarily successful. From the limited evidence, it appears that students will not always take up the offer to receive feedback, or will make limited use of it, following summative written assessments or OSCEs (Sinclair and Cleland 2007; Harrison et al. 2013). Performance appears to be an influential factor, with students who perform very well, and those who fail the assessment, tending to make more use of the feedback, whereas those students who just passed make least use (Harrison et al. 2013).

피드백은 매우 복잡한 과정이지만 학습자의 지식이나 수행 능력을 향상시키기 위해서는 몇 가지 중요한 단계를 협상해야한다는 일반적인 동의가 있습니다 (Hattie and Timperley 2007; Shute 2008; Archer 2010). 

• 첫째, 학습자는 그것을 받아들이기를 수용해야합니다.

• 두 번째로, 학습자는 주어진 메시지를 이해해야하므로 학습자의 Frame of reference와 일치해야합니다 (Kluger and DeNisi 1996). 

• 셋째, 학습자는 구체적이고 의미있고 달성가능한 목표를 설정하고 도달을 위한 조치를 취해야합니다 (Hattie and Timperley 2007, Shute 2008). 

• '피드백 문화'를 창안하는 중요성이 제기되었습니다 (Archer 2010). 최근의 연구는 직업 내의 사회적 문화적 영향이 개인의 감수성 및 피드백에 어떻게 영향을 미칠 수 있는지를 보여주었습니다

Although feedback is a highly complex process, there is general agreement that several important steps need to be negotiated in order for it to be successful in improving the knowledge or performance of the learner (Hattie and Timperley 2007; Shute 2008; Archer 2010). 

• Firstly, a learner needs to be receptive to receiving it.

• Secondly, the learner must understand the message being given, so it must be such that it aligns with the learner’s frame of reference (Kluger and DeNisi 1996). 

• Thirdly, the learner needs to set concrete, meaningful and attainable goals, and then take steps to reach them (Hattie and Timperley 2007; Shute 2008). 

• The importance of creating a ‘culture of feedback’ has been advocated (Archer 2010). Recent work has demonstrated how social and cultural influences within a profession can influence an individual’s receptivity and responsiveness to feedback

피드백 추구 행동은 '가치있는 목표를 달성하기위한 행동의 정확성과 타당성을 결정하기위한 노력의 의식적 헌신'으로 정의 할 수 있습니다 (Crommelinck and Anseel 2013). 

• 첫째, 유용한 정보를 얻을 필요가 있습니다. 

• 둘째로 자기 자신의 에고를 보존하거나 강화할 필요가있다. 

• 셋째, 타인이 그 사람에 대해 가지고 있는 인상을 방어하거나 강화할 필요가있다 (Ashford and Cummings 1983). 

Bok et al. (2013)은 피드백 제공자의 특성, 피드백 탐색자의 특성 및 피드백 탐색자와 제공자 간의 관계가 모두 중요하다는 것을 발견했습니다.

Feedback-seeking behaviour can be defined as ‘‘the conscious devotion of effort towards determining the correctness and adequacy of one’s behaviours for attaining valued goals’’ (Crommelinck and Anseel 2013). 

• Firstly, there is the need to obtain useful information; 

• secondly there is the need to preserve or enhance one’s own ego; and 

• thirdly there is the need to defend or enhance the impression others hold of the person (Ashford and Cummings 1983). 

Bok et al. (2013) found that characteristics of the feedback provider, characteristics of the feedback seeker, and the relationship between the feedback seeker and provider are all important.

현재 연구의 맥락에서, 왜 학습자가 피드백을 탐색하지 않거나, 또는 적극적으로 회피하는지를 연구하는 것이 더 적절할 수 있습니다. 피드백 추구 행동뿐만 아니라, Moss et al. (2003)은 능동 피드백 회피 행동 또한 존재한다고 제안했다. 직장에서 더 열등한 수행자는 자신의 수행 능력이 열악하다는 것을 알고있을 때 적극적으로 피드백을 피할 수 있습니다. 다른 연구들은 또한 poor performer가 피드백을 찾는 경향이 적음을 보여 주었다 (Northcraft and Ashford 1990; Morrison and Bies 1991).

In the context of the current study, it may be more relevant to consider why learners may refrain from seeking, or actively avoid, feedback. As well as feedback-seeking behaviour, Moss et al. (2003) proposed that active feedback-avoiding behaviour also exists. Poorer performers in the workplace may actively avoid feedback when they are aware that their performance is poor. Other studies have also demonstrated that poorer performers are less likely to seek feedback (Northcraft and Ashford 1990; Morrison and Bies 1991).

• 모호함과 불확실성은 학습자의 인지부담을 높여주고 피드백에 대한 반응을 저해 할 수 있습니다 (Shute 2008). 

• 변경해야 할 사항과 유지해야 할 강점에 대한 명확한 지침이 제시된 지침 피드백은 초보자나 struggling 학습자에게 가장 유용하지만 

• 의견과 제안을 포함하는 보다 촉진적인 피드백은 경험이 많은 학습자에게 도움이 된다(Shute 2008). 

• 동시에 매우 복잡한 피드백 메시지는 학습자가 흥미를 잃을 수도 있기 때문에 반드시 도움이되지는 않습니다 (Shute 2008).

Vagueness and uncertainty will lead to a higher cognitive load for learners, which can impair the response to feedback (Shute 2008). Directive feedback, in which clear guidance is given on what needs to be changed and which strengths should be maintained, appears to be most useful for novice or struggling learners, whereas more facilitative feedback, which comprises comments and suggestions, is acceptable for more experienced learners (Shute 2008). At the same time, a very long complex feedback message is not necessarily helpful, as learners may lose interest (Shute 2008).

• 피드백 메시지가 명확하게 이해되는 경우더라도, 학습자는 피드백을 받아 들이고 이를 활용할 필요가 있습니다. 제공된 정보가 이미 학습자가 알고 있는 경우에 학습자는 피드백을 거의 사용하지 않는 경향이 있습니다 (Hattie and Timperley 2007). 

• 신뢰성이 부족한 경우 의견을 무시할 수 있습니다. 예를 들어 피드백 제공자는 학습자를 직접 관찰하지 않은 경우가 있다(Watling 외. 2013). 

• 학습자는 자신의 능력에 대한 자기 평가에 도전하는 피드백 정보를 받는 것에 대해 우려하고 있습니다 (Mann 외. 2011). 

• 비판적 피드백은 강력한 감정적 반응을 유도하여 효과적인 사용을 방해 할 수 있습니다 (Sargeant 외. 2008). 

• 일부 학습자는 피드백을 지식 또는 기술 결함을 교정하기위한 장치라기보다, 자신감을 높이는 수단으로 사용하는 데 더 관심을 보입니다 (Eva 외. 2012).

Even if the feedback message is clearly understood, learners need to be willing to accept the feedback and make use of it. Learners will tend to make little use of feedback if the information provided confirms what they already knew (Hattie and Timperley 2007). Feedback may be disregarded if it is lacking in credibility; for example because the feedback giver has not observed a learner directly (Watling et al. 2013). Learners are apprehensive about receiving feedback information which challenges their own selfassessment of their abilities (Mann et al. 2011). Critical feedback can induce strong emotional reactions which may block effective use (Sargeant et al. 2008). Some learners appear more interested in using feedback as a means of boosting their self-confidence rather than as a device to correct knowledge or skill deficiencies (Eva et al. 2012).

피드백에 대한 개별 학습자의 반응을 고려할뿐만 아니라, Watling et al. (2013) 피드백과 학습이 이루어지는 사회 문화적 배경에 대해 고려해야 한다. 전문직종마다 피드백과 관련하여 조직 문화가 다르며, 이는 학습자가 자신이 받은 피드백이 건설적인지, 그것이 신뢰할만 한지에 대해 인식하는 방식에 영향을 미친다 (Watling et al., 2013).

As well as considering the individual learner’s response to feedback, Watling et al. (2013) advocate consideration of the social and cultural contexts in which feedback and learning takes place. Different professions appear to have different organisational cultures with respect to feedback, which influence how learners perceive the constructiveness and credibility of feedback they receive (Watling et al. 2013).

평가가 학습에 미칠 수있는 영향은 분명히 중요합니다. 이들은 이전에 사전 평가, 사후 평가 및 순수한 평가 효과로 분류되었다 (Dochy et al., 2007). 현재까지 총평 평가의 학습 효과에 관한 연구는 사전 평가 효과에 중점을 두 었으며 사후 평가 효과는 고려하지 않았다 (Rudland et al., Cilliers et al. 2010, 2012, O'Carroll and Fisher 2013).

The effects that assessment can have on learning are clearly important. These have previously been categorised as pre-assessment, post-assessment and pure assessment effects (Dochy et al. 2007). Studies to date on the learning effects of summative assessment have focussed on the pre-assessment effects and not considered the post-assessment effects (Rudland et al. 2008; Cilliers et al. 2010, 2012; O’Carroll and Fisher 2013).

Methods

맥락

Context

The study took place at Keele University School of Medicine in 2011 with students from the third year (out of five). 

• Two-thirds of the way through the year, students have a summative 12-station OSCE, including stations on history-taking, clinical examination and practical procedural skills. Students have to pass at least eight stations in order to progress to the next year. Each station lasted for 8.5 min. New stations are introduced each year to minimise the risk of students ‘studying to the test’. Students have to pass the OSCE, as well as written assessments, in order to progress to Year 4. 

• Following the OSCE, they have a further 12 weeks of study within the year, based in general practice and in selected clinical areas of their choice. 

• Individualised feedback about each student’s performance was released via a website on the same day the results were published but there was no compulsion for students to view the feedback. Students could choose to look at the feedback in different ways; they could view a breakdown of the skills assessed in each station or across the OSCE as a whole. They could also compare their performance with the cohort as a whole. 

More details about this have been published elsewhere (Harrison et al. 2013). A number of screen captures are shown in Appendix 1. In years 1 and 2, the students had received feedback on summative written and practical skills assessments in large group classes. They had not had a formative OSCE earlier in Year 3, but had received more informal teaching regarding preparation for the assessment. 

참여자와 표집

Participants and sampling

• Recruitment of students followed a maximum variation sampling approach to ensure that the sample included students with a wide range of achievement in written and OSCE assessments, and who had used the website in very different ways. Maximum variation sampling helps to achieve information-rich cases for in-depth study (Patton 2002). 

• To preserve anonymity, and in order to avoid any suggestion of coercion, students received an initial email from an administrator, rather than directly from the researchers. Interviewing continued until theoretical saturation was reached.

Individual, semi-structured, face-to-face interviews were conducted in June and July 2011 by the principal researcher (CH), who had previously had little direct interaction with the students. 

• This method was chosen in order to gain in-depth information and encourage deep exploration of personal matters in a way that is often not possible in a group setting (DiCicco-Bloom and Crabtree 2006). 

• The interviewing technique followed recognised methods of rapport development and use of exploratory open-ended questions recommended by DiCicco-Bloom and Crabtree (2006). 

• All the interviews were audio-recorded. 

• The interviews lasted 30–60 min. 

• Questions explored students’ experiences of their recent OSCE, their perceptions of the feedback they received and its impact. 

• The interview protocol is shown in Appendix 2. 

• Themes which emerged in earlier interviews were explored in subsequent interviews.

In total, data were derived from 17 participants. It proved hard to recruit students who had only just passed the assessment (passed 8 or 9 OSCE stations). 

• As these were a group who had been shown to have made little use of feedback, recruitment continued to focus on those students who had accessed the feedback in only a limited manner but had a slightly higher level of performance (Harrison et al. 2013). 

• Details of the participants are shown in Table 1. Ten (59 %) of the interviews were female, which was identical to the proportion of females in the year group as a whole. Twelve (80 %of those who provided information) had entered directly from school which is comparable to the cohort as a whole (73 %). Participants visited the website on one to five separate occasions, viewing between 46 and 316 webpages in 6–70 min. These figures were closely comparable to the cohort as a whole.

자료 분석

Data analysis

Verbatim transcriptions of the interviews were analysed with the assistance of computer-assisted qualitative data analysis software (NVIVO version 9, QSR International Pty Ltd, Doncaster, VIC, Australia). 

• Data were analysed using the constant comparative method, which comprises simultaneous coding and analysis of data in order to develop and refine concepts and explore their inter-relationship (Corbin and Strauss 2008). 

• Analysis combined coding which was guided by a priori awareness of the feedback literature relevant theories with inductive coding which emerged from the data. 

• Findings from the interviews were triangulated, with the participants’ consent, with information from each participant’s usage of the website, and their performance in the assessments (Lingard et al. 2008). 

• The principal researcher (CH) coded all transcripts and constructed initial themes. 

• A quarter of the transcripts were reviewed by a second researcher (VW) against the coding framework and themes. 

• A third researcher (SY), unconnected from the main research project, independently coded two transcripts. 

• Discrepancies were resolved by discussion.

결과

Results

Five themes emerged 

• (1) the power of the summative assessment culture and its negative impact on the use of feedback; 

• (2) the influence of strong emotions; 

• (3) the influence of social interactions with others; 

• (4) the influence of prior learning experiences on expectations; 

• (5) the disconnection between assessment and future learning.

총괄평가 문화의 힘

Power of summative assessment culture

총괄 평가는 실패와 잠재적 처벌에 대한 두려움, 재시 필요성이 지배하는 강력한 문화를 창출했습니다.

Summative assessment created a powerful culture that was dominated by fear of failure and potential punishment, i.e. the need to take re-sits.

실패에 대한 두려움과 평가 장애물을 뛰어 넘어야 할 필요성은 대부분의 인터뷰를 에서 등장하였고, 이런 것이 OSCE가 학습 기회이기보다는 외재적 동기로 간주되게 하였다.

The fear of failure, and the need to jump over the assessment hurdle, pervaded most interviews and prevented the OSCE frombeing regarded as a learning opportunity, but was instead an extrinsic motivator.

이 문화는 역설적으로 평가를 통과 한 학생을 무시하면서, 실패하고 처벌을 요구하는 학생에게 더 많은 관심을 쏟게 했다.

This culture paradoxically focused more attention on students who failed and required the punishment, while ignoring students who passed the assessment:

허들을 장래의 학습을 향한 디딤돌이 아닌 종점endpoint로 여기는 태도는 의대에 의해 강화 된 것으로 인식되었다.

The attitude towards the hurdle being seen as an endpoint, not as a stepping stone towards future learning, was perceived as being reinforced by the medical school.

강력한 감정의 영향

Influence of strong emotions

실패에 대한 두려움은 학생들에게 매우 강렬한 감정을주었습니다. OSCE 당일 시험이 시작되기를 기다리는 것은 불안한시기였습니다. 몇몇 참가자는 신체적으로 아팠습니다. 그 결과를 기다리는 동안 실패에 대한 두려움이 지배적이었고, 이로 인해 더욱 큰 불안이 초래되었습니다. 일부 학생들은 너무 두려워 결과를 열어보기 힘들어했고, 동료 나 친구에게 대신해달라고 요청했습니다.

The fear of failure generated very strong emotions in the students. On the day of the OSCE, the wait for the exam to begin was a time of high anxiety, with several participants being physically sick. Fear of failure was dominant while awaiting the results, which led to even greater anxiety. Some students were too fearful to open the results and asked colleagues or friends to do it on their behalf.

피드백 타이밍이 한 요인이었습니다. 이 경우, 피드백은 다른 서면 시험을 위해 revising하는 동안 release되었으며, 실패한 표식을받는 것에 대한 두려움은 슬픔에 비유되었습니다.

The timing of the feedback was a factor. In this instance it was released while they were revising for other written exams and some did not want it to disturb their revision for these exams, as the fear of receiving a failing mark was likened to a grief reaction:

대조적으로, 성적결과 공개 이후의 기간은, 대다수의 학생들에게 [재시험을 피하는 목표]가 달성됨에 따라 격렬한 안도감으로 표시되었습니다. 이로 인해 학생들이 특정 피드백에 참여할 동기가 줄어 들었습니다. 실패에 대한 두려움에 뒤따라오는 평가 장애물을 극복했다는 엄청난 안도감은 피드백에 대한보다 합리적인 관심을 차단하는 것처럼 보였습니다. OSCE를 수행하는 동안과 결과를 기다리는 동안의 불안감은 안도감을 심화시키는 것으로 보였다.

In contrast, the period after the release of results was marked, in the vast majority of students, by an intense feeling of relief, as the goal of avoiding the re-sits had been achieved. This reduced the motivation of students to engage with the specific feedback. The enormous relief at overcoming the assessment hurdle, following the fear of failure, appeared to block a more rational interest in the feedback. The anxiety, both while performing the OSCE, and while awaiting the results, appeared to intensify the feeling of relief.

다른 사람과의 사회적 상호작용

Social interactions with others

임상 교사, 동료, 부모 및 파트너 (남자 친구 / 여자 친구)와의 상호 작용은 실패를 회피하고자 하는 필요성을 강화 시켰습니다. 학생들은 임상교사가 OSCE를 뛰어넘을 후프로서, 그냥 통과 하는 것 만으로 충분하며, 실제 임상 학습은 별도이고, 그리고 졸업 후 종종 이뤄지는 것이라고 여김을 인식했다.

Interactions with clinical teachers, peers, parents and partners (boyfriends/girlfriends) reinforced the need to avoid failure. Students perceived that clinical teachers regarded the OSCE as a hoop to jump through, for which a pass was sufficient, whereas the real clinical learning took place separately, and often after graduation.

재시험을 볼 필요가 있는지 없는지에 대한 토론을 제외하면, 학생들 간의 피드백에 대한 심층적 인 토론은 거의 없었으며, 이는 학습 피드백을 사용하는 것보다는 Pass-fail 문화에 중점을 두는 태도를 더 강화시켰다. 학생들은 의대를 졸업하고 어딘가에 지원할 때 학업 성적이 사용된다는 점, 그리고 또 부분적으로는 고학업성취자인 의대생의 내재한 경쟁을 인식했기 때문에, 집단 내에서 강한 경쟁적인 태도를 지니고있었습니다. 이 태도는 학생들 사이에서 명확하게 논의되는 경우가 거의 없었으며, 연구참여자들은 왜 그렇게 강하게 존재했는지 설명하는데 어려움을 보였다. 그럼에도 불구하고, 거의 모든 학생들이 숨겨진 경쟁을 감지하고 이것이 피드백에 대한 그들의 반응에 영향을 미치는 것으로 보입니다. 학생들은 종종 자신과 타인의 경쟁에 대한 모순 된 의견을 표현했습니다.

There was rarely any in-depth discussion of the feedback between students, other than discussion as to whether they needed to re-sit or not, which reinforced the focus on the summative pass–fail culture rather than on using the feedback for learning. Students perceived a strongly competitive attitude among the cohort, partly due to academic ranking being used in applications for the first jobs when leaving medical school and partly due to a perceived inherent competitiveness from medical students being high academic achievers. This attitude was rarely explicitly discussed between students and they struggled to explain why it existed so strongly. Nevertheless, almost all students sensed the hidden competition and this appeared to influence their response to the feedback. Students often expressed contradictory opinions about their own and others’ attitudes to competition:

나는 다른 사람들과 비교하여 내가 어떻게하고 있는지를 보는 것이 유용하다는 것을 안다... 모두가 나올 때 '몇 등 했어?'라고 묻는 것이 싫다. ... 나는 나 자신을 보고 싶지만, 그 밖의 다른 모든 사람과 비교하는 것이 싫다

I find it useful to see how I am doing in comparison to everyone else….I hate it when everyone comes out and asks ‘what percent did you get?….I’d like to see myself, but I don’t want everyone else to be comparing’

학생들은 잘 한 경우에도 그것을 드러내는 것이 타인에 의해 오만하게 느껴지거나 uncaring한 것으로 해석 될 수 있다는 두려움 때문에 그 사실을 다른 사람들에게 알려주기를 꺼려했습니다. 어떤 학생들은 학업적으로 아주 뛰어난 학생들을 따라가는 것은 이미 불가능하다고 느꼈고, 경쟁에서 opted out하여 그냥 지나가는 것 만을 목표로 삼았습니다.

Students were reluctant to reveal to others that they had performed well for fear that it would be interpreted by others as arrogant or uncaring towards those who had performed badly. Others felt unable to keep up with the academically strongest performers, and ‘opted out’ of the competition and aimed just to pass.

학생들은 또한 관계를 망치지 않으면서 파트너와도 경쟁해야 하는 복잡한 상황을 협상해야 했습니다.

Students also had to negotiate a complex situation of competition with their partner without upsetting the relationship.

나는 내가 그녀보다 낫다고 생각하는 것처럼 보여지기 싫다. 또한 나는 그녀가 나보다 낫다는 것을 아는 것도 원하지 않는다.

I wouldn’t want to make myself sound like I think I’m better than her. And I wouldn’t want her to make out that she’s better than me, so it works two ways.

학생들이 피드백 데이터를 부모에게 보여줄 의사가있는 것처럼 보였지만, 일반적으로 그 목적은 만족스러운 수준으로 수행하고 있음을 입증하기 위해였습니다. 그들은 심사관보다 부모로부터의 긍정적 인 피드백을 중요시했다.

Although students seemed more willing to show the feedback data to parents, this was usually in order to demonstrate that they are performing to a satisfactory level. They valued positive feedback from parents more than from examiners.

학부모들은 학생들의 태도를 평가에 합격하는 것에 만족시키기 위해 강화했습니다.

Parents reinforced the attitude among students to be content with just passing the assessment.

사전 학습경험이 기대치에 미치는 영향

Influence of prior learning experiences on expectations

의대에 다니기 전에 학생들이 배웠던 문화는 성취에 대한 기대와 개선을 위해 피드백을 사용할 필요성에 영향을 미쳤습니다. 학생들은 학업 성취에 대한 기대치의 변화에 ​​적응하는 데 어려움을 겪었습니다. 학생들은 최상위권에 오르는 것이 익숙했었지만, 지금은 의대생으로서 well down한 상태이다. 이로 인해 최상위권 성적은 달성 할 수 없다는 무용감을 느끼게 했다. 결과적으로, 그들은 너무 열심히 노력하는 것을 피했고 단순히 평가를 통과하는 더 낮은 목표로 만족했습니다. 이것은 평가의 총괄평가적 성격을 더욱 강화시켰다.

The culture in which students had learnt prior to attending medical school affected their expectations about achievement and the need to use feedback to improve. Students reported difficulty in adjusting to a change in their expectations for academic achievement; as school pupils they were used to coming near the top of the class, but now as medical students they were well down the field. This led to a sense of futility at being unable to achieve marks towards the top of the year. As a result, they avoided trying too hard and instead were content with the lower goal of simply passing the assessment. This further reinforced the summative nature of the assessment.

평가와 미래 학습과의 단절

Disconnection between assessment and future learning

직장에서의 임상 학습과 OSCE 간의 단절이있었습니다. OSCE에서 좋은 점수를 얻으려면 임상 현장에서 환자를 보는 데 많은 시간을 할애해야 했던 반면, OSCE의 피드백은 향후 근무지 학습을 위해 필요한 것으로 여겨지지 않았습니다. 이 시험은 특정 시험 (또는 시험 내 특정 질문)과 관련이 있는 것으로 보여졌으며 여러 스테이션의 성능 차이로 인해 여러 스테이션의 일반적인 기술 성능을 고려하지 않았습니다. 미래에 피드백을 사용할 계획이 있다면 이것은 미래의 OSCE와 관련된 것이지, workplace learning을 위한 것이 아닙니다.

There was a disconnection between clinical learning in the workplace and the OSCE. While spending extensive time seeing patients in the clinical workplace was seen as essential in order to gain a good mark in the OSCE, feedback from the OSCE was not seen as necessary for future learning in the workplace. It was seen as relevant only for that specific exam (or for a specific question within the exam) and the variability of their performance across stations made it irrelevant to consider their performance of generic skills across several stations. If there were plans to use the feedback in the future, this was in connection with future OSCEs, not workplace learning.

피드백 결과에 대한 감정적인 반응이 가라앉은 후에는, 피드백을 더 깊이 고려하도록 병원 내 튜터들과 피드백에 대해 논의하는 일은 거의 없었다. 그러나 그렇게 할 경우에는 도움이 되었습니다.

Feedback was rarely discussed with tutors in the workplace to encourage further consideration of the feedback after the emotional response to the release of the results had subsided. When it did take place, this was helpful.

고찰

Discussion

우리의 연구는 또한 학생들이 다른 사람, 특히 부모님, 의 인식을 관리하려고 노력함을 보여주었다. 그 방법은 피드백을 사용하여 자신이 얼마나 잘하고 있는지 보여 주거나 코스의 어려움을 보여 주려고 노력하는 것이었다. 이것은 조직의 심리학 문헌과 약간의 유사점을 가지고 있는데, 직원들은 정보를 얻기 위해 피드백을 사용하기보다는, 다른 사람에게 특정 정보를 강조해서 보여주기 위한 목적으로 사용한다(Ashford and Cummings 1983; Morrison and Bies 1991). 적극적인 피드백-기피 행동의 확실한 증거는 찾지 못했습니다.

Our study has also demonstrated that students seek to manage the perceptions of others, particularly their parents, by using the feedback to demonstrate how well they are doing, or to show how difficult the course is. This has some similarities with the organisational psychology literature, in which employees seek feedback not to obtain information, but rather to highlight information to others (Ashford and Cummings 1983; Morrison and Bies 1991). We did not find clear evidence of active feedback-avoiding behaviour.

학생들이 피드백 메시지를 이해하지 못했다는 증거는 거의 없었지만, 학습자들이 종종 피드백을 사용하도록 동기 부여를 받지 않았다는 것에 대해서는 확증적인 증거가 있었습니다.

Although there appeared to be little evidence that students failed to understand the feedback message, there was confirmatory evidence that learners were often not motivated to make use of the feedback.

이 연구의 결과는 피드백으로부터의 학습이 상황 이론 (Situationivity theory) (Durning and Artino 2011)에 기술 된 바와 같이, 피드백이 일어나는 맥락에서 분리 될 수 없다는 개념을 강화시킨다. 이 이론은 교육자가 학습자에게 정보를 제공하는 방법뿐만 아니라, 정보가 사용되거나 사용되지 않을 상황을 이해하는 데 중점을 두어야한다고 제안합니다. 이론은 또한 authentic 학습 활동이 더 나은 학습을 가져야한다고 제안한다 (Durning and Artino 2011).

The findings from this study reinforce the notion that learning from feedback cannot be dissociated from the context in which it takes place, as described by situativity theory (Durning and Artino 2011). This theory proposes that educators need to consider not just the method by which information is given to learners, but instead focus on understanding the situations in which the information will, or will not, be used. The theory also proposes that authentic learning activities should result in better learning (Durning and Artino 2011).

이 맥락의 다른 중요한 요소로는 다른 학생들과의 사회적 상호 작용 - 그것이 동료 학생이든 교사와 부모의 더 위계적인 영향이든 - 을 포함합니다.

Other important elements of this context include the social interactions with others, whether it be fellow students or the more hierarchical influence of teachers and parents.

함의

Implications for practice

교수진에게 더 많은, 또는보다 우수한 품질의 의견을 제공하도록 요청하는 것은 한계가 있다. 마찬가지로, 학생들에게 그들의 결과와 피드백에 대해 반영하도록 격려하는 것은 거의 영향을 미치지 않을 것입니다. 대신, '피드백 문화'(Archer 2010) 개발에 중점을 두어야합니다. 의료계 내에서 이러한 문화를 달성하는 것은 피드백이 종종 계획된 방식이 아닌 산발적인 방식으로 주어지거나, 직접적으로 관찰 된 성과보다는 추론을 기반으로 할 때 특히 문제가되는 것처럼 보입니다 (Watling 외. 2013).

Asking faculty to provide more, or better-quality, feedback will have only limited success. Similarly, encouraging students to reflect on their results and their feedback is likely to have little impact. Instead, the focus should be on the development of a ‘feedback culture’ (Archer 2010). Achieving this culture within the medical profession seems particularly problematic when feedback is often given in a sporadic rather than planned way, and on the basis of inference rather than directly observed performance (Watling et al. 2013).

Limitations

Suggestions for Further Research

결론

Conclusions

모든 평가는 미래의 학습을 잠재적으로 알리고 향상시킬 수있는 개별 학생에 관한 많은 양의 데이터를 생성합니다. 그러나 통과와 실패에 중점을 둔 총괄평가 문화는 이러한 풍부한 피드백 정보를 사용하는 데 장애가 될 수 있습니다. 그러나 프로그램 수준에서 총괄평가와 형성평가를 분리해도 효과가 없다는 것은 분명합니다. 총괄평가는 의미있는 피드백을 제공하지 않으며, pure한 형성 평가는 장기적으로 덜 심각하게 받아 들여지는 경향이 있습니다. 따라서 학습 프로그램 평가를 위해서는 모든 평가에 형성적 기능과 총괄적 기능을 결합해야합니다.

All assessments generate large amounts of data about individual students which can potentially inform and enhance future learning. However, the culture of summative assessment, with its focus on passing and failing, can act as a disincentive for use of this rich feedback information. Yet it is clear that a separation of summative and formative assessments at the programme level does not work; summative assessments do not provide meaningful feedback and purely formative assessment tends to be taken less seriously in the long run. For an assessment for learning programme therefore, the formative function and summative functions need to be combined in all assessments.

Archer, J. C. (2010). State of the science in health professional education: Effective feedback. Medical Education, 44, 101–108.

Bok, H. G. J., Teunissen, P. W., Spruijt, A., Fokkema, J. P. I., van Beukelen, P., Jaarsma, D. A. D. C., et al. (2013). Clarifying students’ feedback-seeking behaviour in clinical clerkships. Medical Education, 47, 282–291.

Durning, S., & Artino, A. (2011). Situativity theory: A perspective on how participants and the environment can act. Medical Teacher, 33, 188–199.

---

Adv Health Sci Educ Theory Pract. 2015 Mar;20(1):229-45. doi: 10.1007/s10459-014-9524-6. Epub 2014 Jun 7.

Barriers to the uptake and use of feedback in the context of summative assessment.

Harrison CJ1, Könings KD, Schuwirth L, Wass V, van der Vleuten C.

Author information

1

Keele University Medical School, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK, c.j.harrison@keele.ac.uk.

Abstract

Despite calls for feedback to be incorporated in all assessments, a dichotomy exists between formative and summative assessments. When feedback is provided in a summative context, it is not always used effectively by learners. In this study we explored the reasons for this. We conducted individual interviews with 17 students who had recently received web based feedback following a summative assessment. Constant comparative analysis was conducted for recurring themes. The summative assessment culture, with a focus on avoiding failure, was a dominant and negative influence on the use of feedback. Strong emotions were prevalent throughout the period of assessment and feedback, which reinforced the focus on the need to pass, rather than excel. These affective factors were heightened by interactions with others. The influence of prior learning experiences affected expectations about achievement and the need to use feedback. The summativeassessment and subsequent feedback appeared disconnected from future clinical workplace learning. Socio-cultural influences and barriersto feedback need to be understood before attempting to provide feedback after all assessments. A move away from the summativeassessment culture may be needed in order to maximise the learning potential of assessments.

PMID:

 

24906462

 

DOI:

 

10.1007/s10459-014-9524-6