총괄평가가 학습에 영향을 미치는 메커니즘(Adv in Health Sci Educ, 2010)

총괄평가가 학습에 영향을 미치는 메커니즘(Adv in Health Sci Educ, 2010)

The mechanism of impact of summative assessment on medical students’ learning (Adv in Health Sci Educ, 2010)

Francois J. Cilliers • Lambert W. Schuwirth • Hanelie J. Adendorff • Nicoline Herman • Cees P. van der Vleuten

Introduction

I think, as a result of assessment, I know much less than I otherwise would have. (6(V)F65)

It has become axiomatic to refer to the powerful impact of assessment on student learning. Terms like the ‘‘backwash effect’’ (Biggs 1996; Elton 1987, used in general educational literature) and the ‘‘washback effect’’ (Alderson and Wall 1993; Bailey 1996, used in language teaching and testing literature) of assessment, ‘‘consequential validity’’ (Boud 1995), ‘‘test-enhanced learning’’, the ‘‘testing effect’’ or the ‘‘testing phenomenon’’ (Glover 1989; Roediger and Karpicke 2006) and ‘‘test expectancy’’ (Lundeberg and Fox 1991) have been used in this regard.

The impact of assessment on student learning is generally held to be profound. Elton and Laurillard (1979) went so far as to state that ‘‘the quickest way to change student learning is to change the assessment system’’. Boud et al. (1999) state that ‘‘[a]ssessment is the single most powerful influence on learning in formal courses’’. If this is the case, then assessment may well be one of the most powerful tools we have at our disposal to influence student learning. However, even after a almost a century of research, efforts to positively influence learning through assessment do not always yield encouraging results (Gijbels et al. 2009).

Internal-to-programme summative assessment may well exert both a stronger and a more pervasive influence on learning than other assessment practices in higher education (HE), however. As Boud (1995) highlighted, students cannot escape the impact of summative assessment. Given the stakes, the design of such assessment is more typically informed by psychometric than learning considerations and so even if other aspects of assessment in a course have been designed to promote meaningful learning, the impact of summative assessment could trump beneficial effects achieved by other means. Furthermore, more students in HE probably encounter internal-to-programme summative assessment than external-to-programme standardised testing.

As for ‘‘learning’’, it is often not the only phenomenon that authors highlight when writing about the impact of assessment. In HE, the impact of assessment on not just learning but also on non-learning student behaviours intended to enhance marks e.g., ingratiating themselves with lecturers, on student stress and on students’ choice of courses has been highlighted by various authors (Becker et al. 1968; Miller and Parlett 1974; Snyder 1971). From the opposite perspective, assessment is typically identified as one of the contextual factors that impact on learning in models of learning (Biggs 1987; Ramsden 1984; Ross et al. 2003; Vermunt 1996).

There is some descriptive literature focussing on the ‘‘what’’ of the impact of internal-to-programme summative assessment on the process of student learning in HE as Maxwell (2004a: 4) puts it, ‘‘whether x caused y’’ (emphasis in original). In contrast, very little has been written explaining ‘‘how it did so’’ (Maxwell 2004a: 4, emphasis in original). As Bunge (2004: 199) noted, ‘‘any mechanism-free account must be taken to be shallow and therefore a challenge to uncover unknown mechanism(s)’’.

From a process theory perspective, this paper deals with ‘‘events and the processes that connect them’’ (Maxwell 2004b: 248) specifically in one distinctive, internal-to-programme, high-stakes assessment system; thus, the local mechanisms at play in a ‘‘complex network of events and processes in a situation’’ (Miles and Huberman 1994: 146; 147).

외적 동기부여

Extrinsic motivation

평가는 학생의 학습 노력의 양과 분포에 대한 외부의 동기와 영향을 제공합니다. 평가가 있다는 사실만으로도 학생들로 하여금 학습하도록 유도하고 따라서 학습에 소비되는 노력의 양에 영향을 미친다 (Miller and Parlett 1974; Snyder 1971; van Etten 외. 1997). 그러나 평가의 영향이 항상 긍정적 인 것은 아닙니다. 학생들이 점수를 얻기를 희망하지 않거나 뒤늦게 따라 잡을 수 없다고 생각해서 예를 들어 점수가 성공적으로 협상을 할 수 없다고 판단하면 학습을 중단 할 수 있습니다 (Becker 외 1968). 또한 외적 동기 부여에 대한 평가에 대한 반응이 모든 학생에게 균질하지도 않다.

Assessment provides extrinsic motivation and impacts on the amount and distribution of students’ learning efforts. The mere fact of assessment motivates students to learn and therefore influences the quantum of effort expended on learning (Miller and Parlett 1974; Snyder 1971; van Etten et al. 1997). The impact of assessment on effort is not necessarily always positive, however. If students perceive they are unable to successfully negotiate assessment, for example because their marks are so bad they cannot hope to achieve a pass or they are so far behind they believe they cannot catch up, this can result in them stopping learning (Becker et al. 1968). Nor is the response to assessment as extrinsic motivation uniform.

결과

Consequences

평가의 잠재적 결과는 또한 학생 학습에 영향을 미칩니다. 학생들은 배운 내용과 학습 된 내용을 이해하는 것보다 강사의 요구 사항을 충족시키기 위해 무엇을 어떻게 배우는지 적응합니다 (Becker et al., 1968; Ramsden 1984, 1992; Snyder 1971). 이것은 적어도 부분적으로는 그렇게하지 않는 위험이 크고, 자기 존중과 단기 및 장기간의 물질적 이익면에서 일치하는 상당한 보상 (Parlett 1969; Snyder 1971)이기 때문입니다. 주제가 평가에서 다루어질 가능성은 학생들이 학습할 내용을 선택하는데 영향을 미친다 (Becker 등 1968, Becker 등 1961, Miller and Parlett 1974, Snyder 1971, Vermunt 1996) Snyder (1971)가 언급 한 것 '선택적 무시'로 간주합니다. 또한 학생들이 학습 자료에 참여하는 데 철저하게 영향을 미칩니다 (Laurillard 1979, van Etten 외. 1997). 이것은 또한 학생이 선택하도록하는 과제 (Snyder 1971)와 학생들이 과제에 기울이는 노력의 양 (Becker 외 1968; Janssens 외 2002, Struyven et al 2005에 인용)에도 적용됩니다. 학생들이 학습 자료에 참여하는 철저한 방법은 주어진 과제에 대한 성과가 최종 성적을 계산하는 데 기여할 때 영향을받습니다 (Ramsden 1992, Snyder 1971). 구두 평가에서 무지하게 나타날 위험 같은 개인적인 결과 또한 학생들의 철저한 준비에 영향을주었습니다 (Joughin 2007).

The potential consequences of assessment also impact student learning. Students adapt both what and how they learn so as to meet the lecturers’ requirements as manifested in assessment rather than understand the material being learned (Becker et al. 1968; Ramsden 1984, 1992; Snyder 1971). This is at least in part because the risks of not doing so are great, the rewards for conforming, substantial, both in terms of self-esteem and in terms of short and longer term material benefits (Parlett 1969; Snyder 1971). The likelihood of subject matter featuring in assessment impacts on what content students select to learn (Becker et al. 1968; Becker et al. 1961; Miller and Parlett 1974; Snyder 1971; Vermunt 1996) or not—what Snyder (1971) referred to as ‘‘selective negligence’’. It also influences the thoroughness with which students engage with learning material (Laurillard 1979; van Etten et al. 1997). This also holds for assignments students choose to do or not (Snyder 1971) and the amount of effort students devote to tasks (Becker et al. 1968; Janssens et al. 2002, cited by Struyven et al. 2005). The thoroughness with which students engage with learning material is impacted by the contribution that performance on any given assign- ment will make towards the calculation of a final grade (Ramsden 1992; Snyder 1971). Personal consequences like the risk of appearing ignorant in an oral assessment also impacted how thoroughly students prepare (Joughin 2007).

바람직한 결과를 얻기 위하여

Achieving a desired outcome

주어진 학습 결과가 원하는 평가 결과를 가져올 가능성은 학생들의 행동에 영향을 미칩니다. 학생들이 공부하는 데 소비하는 시간은 학습자료의 양과 그것과는 독립적으로 학습자료의 난이도가 증가함에 따라 점차 증가한다 (van Etten et al. 1997). 학생들은 평가 과제의 요구에 가장 잘 대비 한 평가 준비를위한 자원과 활동을 선택합니다 (Frederiksen 1984; Newble and Jaeger 1983). 또한 학생들은 학습 결과를 평가 과제의 요구와 일치시켜 원하는 결과를 얻습니다 (Becker et al., 1968; Sambell and McDowell 1998).

The likelihood of any given learning behaviour bringing about a desired assessment outcome influences students’ actions. The amount of time students spend studying increases, up to a point, as the volume of material and, independent of that, the degree of difficulty of the material, to be studied, increases (van Etten et al. 1997). Students select resources and activities to prepare for assessment that best prepare themfor the demands of the assessment task (Frederiksen 1984; Newble and Jaeger 1983). Student also match the nature of their learning to the demands of the assessment task to achieve a desired outcome (Becker et al. 1968; Sambell and McDowell 1998).

학생들은 학습자가 원하는 결과를 평가 (Becker et al., 1968, 1961) 할 수 있도록 학습하기 위해 내용 선택을 안내하기 위해 신호기, 다른 학생 및 과거 논문을 찾고 심지어이를 끝내기 위해 속일 수 있습니다 (Becker et al., 1968). 많은 양의 작업으로 학생들은 원하는 결과를 얻으려는 목적으로 어떤 내용에 대해보다 선별 적으로 선택하고 낮은 수준의인지 처리 전술을 채택해야합니다 (Ramsden 1984; Snyder 1971; van Etten 외. 1997). 노력은 주어진 시간에 혜택이나 보상이 가장 유용하다고 여겨지는 곳을 기초로 코스에 할당됩니다 (Becker 외 1968)

Students seek cues from lecturers, other students and past papers to guide their selection of content to learn, in the interests of achieving their desired outcome with assessment (Becker et al. 1968, 1961), and may even cheat to achieve this end (Becker et al. 1968). High volumes of work drive students to be more selective about what content to engage with and to adopt low level cognitive processing tactics in the interests of achieving a desired outcome (Ramsden 1984; Snyder 1971; van Etten et al. 1997). Effort is allocated across courses based on where generating benefit or reward is deemed most useful at any given time (Becker et al. 1968).

목표

Goals

학생들의 목표는 평가에 대한 그들의 반응에 영향을 미칩니다. 학생들은 자신이 원하는 수준의 학업 성취도를 측정합니다 (Becker et al., 1968; Miller and Parlett 1974; van Etten 외. 1997). 학생들이 주제를 향상시킬 필요가 있는지 여부, 자료가 흥미로운 지 여부, 자료가 다루기가 쉽지 않은지 (예 : 이해하기가 불가능하지 않은지 여부) 및 과제가 주요 연구 영역에 있는지 여부 (van Etten et al., 1997). 흥미롭게도, 학생들이 인식하는 평가 요구를 충족시키기 위해 채택하는 학습 유형은 장기 목표에 따라 달라질 수 있습니다.

Students’ goals influence their response to assessment. Students gauge the magnitude oftheir efforts by what grade they aimto achieve (Becker et al. 1968; Miller and Parlett 1974;van Etten et al. 1997). Various factors influence the priority students accord reading assignments, including 

• whether they need to improve in the subject, 

• whether the material is interesting, whether the material is manageable (e.g., not impossible to understand) and 

• whether the assignment is in their major area of study (van Etten et al. 1997). 

Interestingly,the type of learning students adopt to meet the perceived demands of assessment may be discordant with their long-term goals.

규범

Norms

평가에 대한 개별 반응은 동료 집단 내에서 사회적으로 구성되고 공유 된 기준 틀에 의해 조절 될 수있다 (Becker et al., 1968). 피어 그룹 내의 규범은 학생이 학습을 시작할 때 조절할 수 있으며, 그렇지 않으면 나중에 선택하기 시작합니다 (Thomson and Falchikov 1998).

Individual responses to assessment can be modulated by a socially constructed and shared frame of reference within a peer group (Becker et al. 1968). The norms within a peer group can modulate when a student starts learning, resulting in them starting later than they would otherwise have chosen to (Thomson and Falchikov 1998).

에이전시

Agency

학습에 대한 평가가 학습 성과에 영향을 미칠지에 대한 학생들의 신념은 학습 동기에 영향을 미칩니다 (van etten 외. 1997). 학생들이 학습을 시작할 때, 주어진 작업량을 고려할 때, 주어진 지각과 복잡성에 대한 대처 능력에 대한 그들의 인식에 영향을 받는다 (Snyder, 1971).

Students’ beliefs as to whether studying would influence their performance on assessments affects their motivation to learn (van Etten et al. 1997). When students start learning is influenced by their perception their ability to cope with a task of given magnitude and complexity, given the prevailing workload (Snyder 1971).

감정

Emotion

Fransson (1977)은 또한 학습에 대한 학생들의 접근 방식은 평가와 관련된 두 가지 요소 인 위협과 불안의 정도에 영향을 받음을 보고했습니다. 평가에 대한 걱정은 학습 노력 배분에 영향을 미치는 것으로보고되었다 (Miller and Parlett 1974).

Fransson (1977) also reported students’ approach to learning is impacted by the degree of threat and anxiety they experience, both factors associated with assessment. Worry about assessment has also been reported to influence allocation of effort to learning (Miller and Parlett 1974).

기존 문헌에서 두 가지 사실이 두드러진다. 첫 번째는 대부분이 연구가 학습에 대한 평가의 영향을 체계적으로 조사하도록 고안되지 않았기 때문입니다. 학습에 대한 평가의 영향에 대한 증거를 제공하는 것으로 종종 인용되는 다양한 연구가 있습니다. 그러나 이들 중 많은 것들이 생태 학적 타당성이 제한된 통제 된 환경에서 수행 된 실험적 작업이었다 (Lundeberg and Fox 1991). 또한, 이러한 많은 연구는 초등 학교를 포함한 학교 환경에서 실시되었으므로 HE 환경에서의 유용성을 더욱 제한하고 있습니다.

Two things about this literature are striking. The first is that, for the most part, these studies were not designed to systematically investigate the impact of assessment on learning. There are various studies that are often cited as providing evidence of the impact of assessment on learning. However, many of these were experi- mental work conducted in controlled settings with limited ecological validity (Lundeberg and Fox 1991). Furthermore, many of these studies were conducted in school settings, including elementary schools, further limiting their usefulness in HE settings.

이 문헌에 두드러진 두 번째 점은 이론적 틀 내에서 평가의 영향을 설명하기위한 제한된 시도 만이 이루어 졌다는 점입니다.

The second thing that is striking about this literature is that only limited attempts have been made to explain the impact of assessment within a theoretical framework.

방법

Methods

맥락

Context

This study was conducted at the Faculty of Health Sciences of Stellenbosch University in South Africa. Medical students there follow a 6 year, modular program.

대상 및 윤리

Subjects and ethics

Thirty-two students volunteered for interviews.

Some characteristics of respondents are summarized in Table 1.

자료 수집 및 분석

Data collection and analysis

In-depth, unstructured interviews (Charmaz 2006; DiCicco-Bloom and Crabtree 2006; Kvale 1996) were conducted with individual students, each lasting approximately 90 min. In keeping with the inductive nature of the study, no formal interview schedule was used. Interviews were loosely constructed around exploring three issues: how respondents learned, what assessment they had experienced and how assessment had impacted on their learning. Open-ended questions were used and statements respondents made were probed to clarify meaning, obtain additional detail and ascertain what assumptions underlie them. For example, vague statements like ‘‘I learn differently for long questions and multiple choice questions’’ were probed for detail about what respondents did differently in the two situations and why they did so.

Although the interviews were conducted at one point in time, students’ experience of different assessment methods and how they learned in varying contexts across all of their years of study were explored, compared and contrasted during interviews, though typically not chronologically. This revealed qualitative and quantitative differences and changes in respondents’ learning across varying assessment contexts and time. Each interview was

allowed to develop its own direction within the broad three-topic framework, so as to allow in-depth exploration of each respondent’s experiences and conceptions of the relationships being studied. Given that data collection proceeded in tandemwith, and was later informed by, data analysis, as analysis proceeded, emerging constructs were also discussed with respondents to confirminterpretation and explored in greater depth in subsequent interviews.

All interviews were conducted by the same investigator, an educational adviser involved in curriculum development in the faculty with little direct student interaction, but much interaction with lecturers. All interviews were conducted in a setting suggested by respondents. Interviews were conducted in either English or Afrikaans, according to respondents’ preference. Care was taken to alert respondents to the fact that their personal accounts were of interest, so that they recounted their own experiences and views rather than what they may have perceived the interviewer to want to hear. Several respondents had to be encouraged to relate their personal experiences and approaches ‘‘warts and all’’, rather than their sanitized impressions of how they thought they should be learning or of how they perceived the nebulous ‘‘they’’ (i.e., other students) to approach learning and assessment. Despite being given an undertaking regarding the confidentiality of data at the start of each interview, several respondents also had to be reassured during their interview about the confidentiality of their comments, before they proceeded to share information they perceived could elicit unfavourable responses from the lecturers concerned. That said, almost all interviews ‘‘caught fire’’ and had to be carefully kept on track as respondents enthusiastically discussed the topic at hand.

All interviews were audio recorded and transcribed verbatim, to ultimately generate almost 1,000 pages of transcripts. Data analysis commenced even as data collection pro- ceeded. Before progressing to more detailed analysis, field notes were reviewed and each transcript was read to obtain a global impression of how assessment impacts on student learning. Initial open coding was then undertaken by one of us (FC). As data collection and analysis progressed, codes were developed, refined and revised in an iterative process (Charmaz 2006; Dey 1993; Miles and Huberman 1994). Ongoing data collection, com- parisons of codes within and between interviews and discussions between team members served to confirm and clarify codes. Clustering and partitioning of codes led to the emergence of categories as data analysis progressed, which categories were also iteratively refined, revised, discussed and ultimately related to one another.

As analysis progressed and relationships between constructs became more established, it became evident that various dimensions of motivation and emotion featured prominently when exploring the link between assessment and learning. Focussed coding of the existing dataset at that point was undertaken. However, while confirming a role for motivation and emotion, this proved to be an inadequate explanatory framework. In many instances, it was simply not possible to label a mechanism by which assessment exerted an influence on learning using this framework. Despite extensive efforts re-appraising existing data and exploring constructs in subsequent interviews, no further useful constructs could be dis- cerned. In fact, nothing new emerged during data collection subsequent to interview fourteen, despite the individualized nature of each interview and adaptations that were made on the basis of preliminary data analysis. Analysis stalled at this point, it being apparent that a framework was needed that transcended motivation and emotion.

결과

Results

Respondents’ learning behaviour was influenced by appraising the impact of assessment, appraising their learning response, by their perceptions of agency and by contextual factors (Fig. 1).

영향력에 대한 판단

Appraisal of impact

응답자는 평가의 영향과 관련하여 두 가지 요인, 즉 발생할 가능성이 얼마나 될지와 결과의 크기가 어느 정도인지를 고려했습니다.

Respondents considered two factors relating to the impact of assessment: how likely consequences were to accrue and what the magnitude of consequences was likely to be.

Likelihood of impact

Magnitude of impact

반응에 대한 판단

Appraisal of response

평가를 심사숙고 할 때 응답자는 특정 결과, 응답의 비용 및 응답자의 개인 목표 및 성공과 건강에 대한 개념에 대해 측정 한 응답의 가치를 달성 할 때 주어진 학습 반응의 효용성을 다양하게 고려했습니다. 평가에 대한 학습 반응은 일반적으로 고립되어 고려되지 않았으며 응답자의 삶의 다른 차원에 대한 요구와 관심과 균형을 이루었습니다.

When contemplating assessment, respondents variously considered the efficacy of any given learning response in achieving a particular outcome, the costs of that response and the value of that response as measured against the respondent’s personal goals and their conceptions of success and wellness. The learning response to assessment was typically not considered in isolation, but rather balanced against demands from and interests in other dimensions of respondents’ lives.

Response efficacy

Response costs

Value attached to expected outcome

자기효능감에 대한 인식

Perceived self-efficacy

자기 효능감은 역경에도 불구하고 상황에 대해 어느 정도 통제력을 발휘할 수 있다는 인식과 관련이 있습니다. 응답자들은 주어진 시간대에 학업 적으로 성취 할 수있는 것을 시간이 지남에 따라 개발하고 평가 될 때 미리 정해진 목표를 달성하기위한 학습 노력의 규모, 분포 및 성격을 조정할 수 있다고보고했습니다 (인용 4, 6).

A sense of self-efficacy has to do with the perception of being able to exert some control over a situation, even in the face of adversity. Respondents reported developing a sense over time of what they were able to achieve academically in any given time frame, and being able to calibrate the magnitude, distribution and nature of their learning efforts to achieve their predetermined goals when being assessed (cf. Quotes 4, 6).

맥락적 요인

Contextual factors

응답자의 맥락에서 가장 중요한 맥락 적 요인은 다양한 referent이었다. Referent은 개인의 가치관이 의도와 행동에 영향을 미치는 데 중요한 역할을하는 사람들입니다. 그들은 동기 부여가된다면 개인이 자신의 행동을 조정할 수있는 규범 적 신념을 제공합니다. 응답자의 경우 주요 referent 그룹은 강사 및 기타 학생들이었습니다. 강사는 직접적으로나 간접적으로 referent이되었다. 학생들은 응답자의 동료와 이전에 특정 학습 연도를 성공적으로 협상 한 학생을 모두 포함시킬 수 있습니다. 일부 응답자는 밀러와 파렛 (Miller and Parlett, 1974)이 기술 한 것과 같이 큐 레이서 (cue seeker)라는 범주에 속하게되었습니다. 다른 사람들은 큐 의식이 있었고 더 많이 그들이 자신이 있다고 느끼는 문제의 정도가 커졌습니다.

The most important contextual factors in respondents’ context were various referents. Referents, people whose opinion an individual values, play an important role in influencing intent and behaviour. They provide normative beliefs against which an individual can calibrate their behaviour, if so motivated. For respondents, two key groups of referents were lecturers and other students. Lecturers served as referents both directly and indirectly. Students could include both peers of the respondent and students who previously success- fully negotiated the particular year of study. Some respondents clearly fell in the category of cue seekers as described by Miller and Parlett (1974). Others were cue conscious and became more so the greater the degree of trouble they perceived themselves to be in.

Normative beliefs

Motivation to comply with normative beliefs

Negative cases

고찰

Discussion

이 연구에서 설명한 요인들이 개입의 단순한 목표가 되지 못합니다. 주어진 시간에 주어진 모든 학생에 대해 이러한 요인들이 모두 작용하는 것은 아니며, 시간 및 상황에 따라 주어진 학생의 일정한 요소에 대한 요인도 아닙니다. 두 학생의 동일한 요소가 작용한다고하더라도, 그 영향의 강도는 개인적 또는 다른 상황에 따라 달라질 수 있습니다. 이것은 "현상의 사회적 및 문화적 맥락이 인과 적 메커니즘의 작동을 이해하는데 결정적이다"(Maxwell 2004a : 6)는 것을 강조한다. Gebhardt와 Maes (2001)가 건강 행동의 맥락에서주의하는 것처럼, 모든 행동이 행동을 유도하는 요인들에 대한 고려 된 반응의 결과 인 것은 아니다. 따라서 학습에 영향을 미치기 위해 평가가 조작되면 학생은 처음에는 고의적 인 변화를 고려하기 위한 노력을하기보다는 확립 된 행동 패턴을 취할 수 있습니다. 이것은 신중하게 설계된 평가 개입의 바람직한 영향력 부족을 설명하는 부분적 방법이 될 수 있습니다.

The factors described in this study do not form a simple target for intervention. Not all of these factors are in play for any given student at any given time, nor are the factors in play for any given student constant across time and context. Even if the same factors are in play for two students, the intensity of that impact may vary based on personal or other contextual influences individual to each. This serves to underline that ‘‘the social and cultural contexts of the phenomenon studied are crucial for understanding the operation of causal mechanisms’’ (Maxwell 2004a: 6). To complicate matters even further, as Gebhardt and Maes (2001) caution in the context of health behaviour, not all behaviour is the result of a considered response to the factors inducing the behaviour. Hence, when assessment is manipulated to influence learning, students may initially act out established patterns of behaviour rather than making the effort to make considered, deliberate changes. This might go part way to explaining the lack of desirable impact of thoughtfully designed assessment interventions.

또한 평가는 전형적으로 학생들이 반응하는 단일한 동질적 존재가 아니라는 점을 강조합니다. 각 모듈 또는 코스는 종종 다른 학생들과 동기화 된 방식보다는 독립적으로 학생들의 요구를합니다. 궁극적으로 가능한 한 많은 학생들을 대상으로 여러 차원의 학습에 영향을 미치기 위해서는 개별 평가 이벤트를 조정하는 것보다 평가 시스템 즉, 평가 시스템을 다룰 필요가 있습니다. 그러나 이러한 복잡성을 고려하지 않으면 평가를 통한 학습에 긍정적 인 영향을 미치는 노력이 실패하게됩니다.

It also bears emphasising that assessment is typically not a single, homogeneous entity to which students respond. Each module or course makes its own demands of students, often independently of rather than in synchronised fashion with others. Ultimately, to have an impact on as many dimensions of learning of as many students as possible, it will be necessary to manipulate multiple dimensions of assessment i.e., assessment systems, rather than tweak individual assessment events. Yet not taking this complexity into account will result in failed efforts to positively influence learning using assessment.

학습에 영향을 미치기 위한 평가는 결과를 보여야하며, 학생들의 표식과 진행에 대한 결과 또는 다른 요인들, 예를 들면 프로젝트 발표 나 동료 평가 또는 대행 감각의 경우 동료 학생의 눈에 띄는 존중 피드백의 경우처럼

Assessment that is intended to impact learning should have consequences, be those consequences on students’ marks and progression or on other factors e.g., their esteem in the eyes of fellow students as may be the case with project presentations or peer assessment or their sense of agency as may be the case with feedback.

어떤 평가 활동의 영향도는 아마도 그것과 관련된 결과의 심각성의 정도와 강하게 상관 될 것이다. 학생의 학년 중 10 %를 기여하는 독창적 인 과제에 대한 피드백을 도입하면인지 적 과제의 수준을 리콜에서 문제 해결로 변경하는 것보다 덜 성공적인 개입이 될 가능성이 높습니다. 객관식 평가에서 50 % 학생의 학년 중.

The degree of impact of any assessment activity will probably be strongly correlated with the magnitude of the severity of consequences associated with it. Introducing feed- back on a one-of-a-kind assignment contributing 10% of a student’s grade is likely to be a less successful intervention than changing the level of cognitive challenge from recall to problem-solving in a multiple choice assessment contributing 50% of a student’s grade.

평가는 학생들이 학습 반응의 효능을 평가할 때 평가가 학계에서 그들이 배워야한다고 생각하는 방식으로 학습하도록 유도해야합니다. 분명히 Newble and Jaeger (1983)와 Frederiksen (1984)은 이러한 변화를 가져올 수있었습니다. 이론 모듈에서는 길고 짧은 질문 유형의 혼합 백을 사용하면 학생들이 학습 내용과 학습 방법 측면에서 학습 자료와 다르게 참여해야합니다. 그들은 물어볼 질문의 길이 나 유형에 따라 단순히 일을 생략 할 수 없습니다. 평가 당 한 번 더 긴 질문 (예 : 20 점)을 짧은 질문과 섞으면 학생들의 학습에서 질적 인 차이가 발생합니다. 분명히, 그런 디자인은 심리 측정 적 및 정치적 고려 사항을 포함하여 다양한 다른 것에 대해서 또한 무게를 달아야 할 것이다 (van der Vleuten 1996).

Assessment should be designed in such a way that when students make their appraisal of the efficacy of their learning response, that appraisal leads them to learn in ways we as academics believe they should be learning. Evidently, Newble and Jaeger (1983) and Frederiksen (1984) were able to bring about just such changes. In theory modules, using a mixed bag of longer and shorter questions types would force students to engage differently with learning material, both in terms of what they learn and in terms of how they learn. They would be unable to simply leave out work based on the length or type of questions to be asked. Having one longer (e.g., 20 mark) question per assessment mixed with shorter questions should lead to qualitative differences in students learning. Clearly, such a design would also have to be weighed against various other, including psychometric and prag- matic, considerations (van der Vleuten 1996).

주어진 학습 반응에 대한 학생들의 비용은 너무 높아서는 안됩니다. 평가 시스템의 요구가 너무 부담 스럽기 때문에 비용 편익 분석을 통해 학생들은 지름길을 찾을 수 있습니다. 여기서 평가를위한 포트폴리오의 도입을 생각합니다. 이것은 건전한 교육 근거를 기반으로하지만 학습에 항상 유익한 영향을 미치지는 않습니다 (Driessen 외. 2007).

The costs to students of any given learning response should not be too high. Where the demands of an assessment system become too onerous, the cost-benefit analysis will lead students to find short cuts. One thinks here of the introduction of portfolios for assessment. This is based on sound educational grounds, but does not always lead to a salubrious impact on learning (Driessen et al. 2007).

마찬가지로, 학생들이 알 수 없거나 복잡하거나, 너무 복잡한 자료 (van Etten 외 1997)에 기반하거나, 학술 모듈 내에서 제한된 시간 내에 의미있게 참여하기에는 너무 방대하다면, 학생 학습에 긍정적 인 영향을 미치지 않을 것입니다.

Equally, assessment tasks that challenge students’ sense of agency by virtue of being unknown or complex, or based on material too complex (van Etten et al. 1997) or too voluminous to engage with meaningfully in the limited time available within an academic module, will be unlikely to have a positive impact on student learning.

마지막으로, 모듈 및 / 또는 강사 주위에 소용돌이 치는 평가에 대한 '신화 및 전설'의 영향은 시야에서 사라져서는 안됩니다. 과거 세대의 학생들에게서 다음 세대의 학생들에게 과거의 시험 서류가 전달되지 않는 대학은 실제로는 드물다. 이들은 많은 학생의 학습을 알리는 내용 및 스타일 분석의 기초를 형성합니다. 따라서, 그들은 그림자에서 벗어나 평가 시스템에 공개적으로 통합되어야합니다.

Finally, the impact of the ‘‘myths and legends’’ about assessment that swirl around a module and/or a lecturer should not be lost from sight. The university where no volumes of past examination papers are passed from one generation of students to the next is likely a rare place indeed. These form the basis of analyses of content and style that inform the learning of many a student. As such, they should be brought out of the shadows and incorporated openly into the assessment system.

또한이 모델은 총괄평가의 맥락을 넘어서는 유용 할 것으로 기대된다. 피드백을 사용하는 수수께끼 중 하나는 잠재적으로 학생 학습에 강력한 영향을 미치는 것으로 밝혀졌지만 (Black and Wiliam1998), 종종 학생들이 놓치거나 잘못 이해하게됩니다 (예를 들어 Gibbs and Simpson 2004 참조). 피드백에 대한 연구 결과의 일부에 영향 평가 및 반응 평가 렌즈를 적용하면 이러한 현상을 밝힐 수 있다고 추측하는 것은 유혹스러운 일입니다. 이 연구에서 강사의 피드백이 학습에 영향을 미치는 요인으로 작용하지 않았다는 점은 흥미 롭습니다.

It is also anticipated that this model might be useful beyond the context of summative assessment. One of the conundrums with using feedback is that whilst it has been found to potentially have a powerful impact on student learning (Black and Wiliam1998), it is often missed or misunderstood by students (see Gibbs and Simpson 2004 for examples). It is tempting to speculate that applying the lens of impact appraisal and response appraisal to some of the findings of research on feedback could shed some light on these phenomena. It is interesting to note that feedback from lecturers did not feature as a factor influencing learning in this study.

램스 덴 (Ramsden, 1992)은 다음과 같이 지적했다. "부적절한 평가 방법은 학생들에게 잘못된 학습 접근법을 강요하는 불가항력적인 압력을 부과한다." 평가가 학생 학습에 미치는 영향에 대해 더 잘 이해하면 의미있는 학생 학습을 유도하고 이러한 상황을 개선하기 위해 평가를 더 잘 활용하는 방법을 모색하는 것이 가능할 것으로 기대됩니다. 평가가보다 효과적으로 활용되도록하는 중요한 링크에는 학생들을 평가하는 학계와 학계가 책임 져야하는 요구 - 평가 - 관련 및 기타 사항의 혼합을 점점 더 결정하는 관리자가 포함됩니다. 더 큰 학문적 기계에서 이러한 톱니 바퀴를 이해하면 앞으로 수십 년 동안 학습에 대한 평가의 비통 한 영향을 아직도 애도하지 않고 있음을 확인할 수 있습니다.

As Ramsden (1992) pointed out: ‘‘Unsuitable assessment methods impose irresistible pressures on a student to take the wrong approaches to learning tasks’’. With a better understanding of how assessment impacts on student learning, it will hopefully be possible to start exploring how assessment can be better utilized to bring about meaningful student learning and remedy this situation. Crucial links in ensuring that assessment is utilized more effectively will include the academics who assess students, and administrators who increasingly decide on the mix of demands—assessment-related and otherwise—to which academics should be answerable. Understanding these cogs in the greater academic machine will hopefully ensure that we are not still lamenting the deplorable impact of assessment on learning some decades hence.

---

Adv Health Sci Educ Theory Pract. 2010 Dec;15(5):695-715. doi: 10.1007/s10459-010-9232-9. Epub 2010 May 9.

The mechanism of impact of summative assessment on medical students' learning.

Cilliers FJ1, Schuwirth LW, Adendorff HJ, Herman N, van der Vleuten CP.

Author information

1

Stellenbosch University, Stellenbosch, Western Cape, South Africa. fjc@sun.ac.za

Abstract

It has become axiomatic that assessment impacts powerfully on student learning, but there is a surprising dearth of research on how. This study explored the mechanism of impact of summative assessment on the process of learning of theory in higher education. Individual, in-depth interviews were conducted with medical students and analyzed qualitatively. The impact of assessment on learning was mediated through various determinants of action. Respondents' learning behaviour was influenced by: appraising the impact of assessment; appraising their learning response; their perceptions of agency; and contextual factors. This study adds to scant extant evidence and proposes a mechanism to explain this impact. It should help enhance the use of assessment as a tool to augment learning.

PMID:

 

20455078

 

PMCID:

 

PMC2995206

 

DOI:

 

10.1007/s10459-010-9232-9