그룹활동에서 "나" 평가하기: 현황과 권고(Med Teach, 2016)

그룹활동에서 "나" 평가하기: 현황과 권고(Med Teach, 2016)

Assessing the ‘‘I’’ in group work assessment: State of the art and recommendations for practice

Joost Dijkstraa,e, Mieke Latijnhouwersb,e, Adriaan Norbartc,e and Rene A. Tiod,e

aMaastricht University, Maastricht University Office, Maastricht, the Netherlands; bRadboud University Medical Center Nijmegen, Radboudumc Health Academy, Nijmegen, the Netherlands; cLeiden University Medical Center, Leiden, the Netherlands; dDepartment of Cardiology and Center for Educational Development and Research in Health Professions, University Medical Center Groningen, Groningen, the Netherlands; eSpecial Interest Group on Assessment of the Netherlands Association for Medical Education (NVMO)

Introduction

의대생은 전문가가되기 위해 훈련을받으며 팀에서 함께 일해야합니다. 따라서 의학 교육에서 '협력자collaborator’’ '의 역량 역할을 다루는 것이 중요합니다 (Frank et al 2015).

Medical students are trained to become professionals, who must work together in teams. It is therefore important to address the competency role of ‘‘collaborator’’ in medical education (Frank et al. 2015),

• 집단 작업 평가Group work assessment는 이 역량을 평가하는 가장 보편적 인 방법이며 (Epstein & Hundert 2002), GW는 학습에 여러 가지 이점이있다. 깊이 있고 적극적인 학습 (Davies 2009), 지식 성과, 팀워크 기술 및 상호 작용 향상 (McMullen 외 2014) 및 직원 및 학생 만족도 향상 (Zgheib 외. 2010).
  
  Group work assessment is the most common way of assessing this com- petency (Epstein & Hundert 2002) and Group work has multiple advantages for learning. It leads to deep and active learning (Davies 2009), increased knowledge outcomes, teamwork skills and interactivity (McMullen et al. 2014) and staff and student satisfaction (Zgheib et al. 2010).

GW평가에서 그룹 전체는 종종 그룹 작업 (예 : 종이, 발표, 포스터)의 결과 인 그룹 제품에 대해 단일 등급을받습니다 (Cheng & Warren 1999) .
In group work assessment, the group as a whole often receives a single grade for a group product, which is the out- come of the group work—for example, a paper, a presenta- tion, a poster (Cheng & Warren 1999).

 

그룹 프로세스가 면밀히 모니터링되지 않고 개별 학생의 기여가 확인되지 않은 경우 (Watson et al., 1993), 개별 학생의 그룹 점수의 유효성에 대한 문제가 제기 될 수 있습니다. 무임승차자는 그룹 활동에 노력을 기울이지 않지만 다른 사람들의 과업에서 과도하게 이익을 얻기를 희망하는 학생으로 정의됩니다. 질문은 다음과 같습니다 : 우리는 그룹 작업에서 개별 구성 요소를 확인하고 이를 평가 기준에 포함시킬 수 있습니까? 그룹과제을 평가 할 때 accountability에 대한 걱정이 생기는 이유는 주로 개인 기여가 평가되는 방법이 불분명하기 때문이다
When group processes are not closely monitored and contributions of individual students not identified (Watson et al. 1993), the validity of group scores for individual students may be challenged. Free riders are defined as students who do not put effort into group work but hope to benefit excessively from the work of others. The question is as follows: can we identify the individual component in group work and include this in the assessment criteria? Worries about accountability arise when dealing with group assignments, mainly because it is often unclear how individual contributions are assessed.

Methods

주요 포함기준

Main inclusion criterion:

•   Assessment of the individual component of group work is described

추가 포함기준

Additional inclusion criteria:

•   The type of group work is described in sufficient detail

•   Grading/judgment procedures/criteria are described in sufficient detail

•   Publication in English

 

 

Results

Thirteen articles met our inclusion criteria. During an additional citation search, we identified one (Spatar et al. 2015) that cited several of the articles selected and fitted all our inclusion-criteria. We included this paper and ended with a total of 14 articles.

The results from the literature review and the question- naire are presented regarding tools, criteria, and procedures, respectively. Characteristics of the 14 selected studies are described and summarized in Table 1.

 

 

평가도구 Tools

동료평과와 자기평가를 대부분 같이 사용

In most studies, the individual component was assessed using peer and/or self-assessment: 12 studies used peer assess- ment, seven of which in combination with self-assessment.

교사가 관찰하지 못하는 부분을 평가하기 위해서 뿐만 아니라, 동료를 평가하면서 배울 수 있게 하려는 교육적 이유도 있다.

Evaluation by peers was not only utilized to assess aspects that cannot be observed directly by teachers (not-ably collaboration in the group) but also for educational reasons, as students learn through the evaluation of the assignments of peers. 

평가준거 Criteria

 

 

process 또는 product가 별도의 기준으로 사용되곤 하지만, 더 흔한 경우는 ‘‘contribu-tion to the group work.’’ 라는 식으로 명명되는 것이었다. 그러나 이 'contribution'이 무엇인지는 잘 정의되지 않았다.

In our review, we found that in peer assessment, process or product were seldom used as separate criteria to evaluate individual students but more often framed as the ‘‘contribu-tion to the group work.’’

• This concept of contribution was poorly defined in eight of the 14 studies.

• The other six used well-described criteria or rubrics regarding the group pro-cess.

• The study by Lejk and Wyvill describes a set of six cri-teria plus keyword indicators (Lejk & Wyvill 2001, 2002)

• that is also used by Sharp (2006), such as motivation, adaptabil-ity, creativity, communication skills, general team skills, and technical skills.

• Strom et al. describe a set of 25 criteria on collaboration skills (Strom et al. 1999).

• In the remaining 10 studies, students were asked to judge the contributions in a more holistic manner.

holistic judgment 가 준비과정에서 이뤄지기도 한다. 학생에게 출석/노력/책임감 등을 성찰하게끔 시키기도 한다.

This holistic judgment was sometimes preceded by some preparation by the students.

• Students were, for example, instructed to reflect on a set of behavior-related questions, for example, concerning peer attendance,effort, responsibility (Dingel & Wei 2014).

• Another way of assessing individual contributions is described by Tucker who used a validated instrument using specific and well-described aspects of group work combined with a more holistic approach (Tucker 2013). 

다양한 사례들

• In only one study, specific teamwork skills were described and used for individual assessment (Strom et al.1999).

• In the wiki study by Caple and Bogle (2013), specific aspects of the process were assessed using the Wikispace platform: a History tab revealed the evolution of the page over the duration of the project (and the student respon-sible for each edit); and the Wiki Statistics function collated every contribution/edit made by an individual member(Caple & Bogle 2013).

• In the study by Kuisma (2007), a port-folio was used for individual grading, and hence, in this case, only reflection on own learning and no peer assess-ment was used.

• The content of the portfolios was graded using the SOLO taxonomy (Biggs & Collis 1982).

• Finally, in one study, explicit criteria for evaluating the end product, a presentation, were mentioned. These, as well as a weighting scheme were negotiated with the class (Knight 2004). 

그룹 과제에 추가로 개인별 과제를 주고, 이를 기반으로 평가하기도 한다. 예를 들면 결과물 발표의 일부를 담당하게 하여, 개개인이 발표와 관련하여 질의응답 하도록 함.

Other ways to identify an individual component mentioned were based on assessing an additional individual task related to the group assignment. For example, being responsible for a part of the presentation of results of the group work, or individually answering questions regarding the presentation.

Box 1. Recommendations for group work assessment*

1. GW에 적합한 과제 Develop tasks that are suited for group work

•    collaboration is beneficial for the result (big enough tasks e.g.)

•    related to a collaborative process in professional practice (realistic/ authentic) 

2. 그룹 구성에 관심갖기 Pay attention to group composition

•    limited group size may diminish the risk of free riding 

3. 학습목표와 평가기준에 '협력'을 넣기 Incorporate collaboration in the learning objectives and assessment criteria 

4. 능숙한 교사가 '그룹 프로세스'에 관심을 갖고 가이드하기(결과에만 집중하지 않기) Provide attention and guidance to the group process by skilled and experienced teachers (do not focus merely on the resulting product) 

5. 그룹 프로세스를 '정기적'으로 평가하기 Evaluate the group process periodically, not only after finishing the task 

•   give opportunity to use feedback to improve group process and product 

6. GW의 평가에서 개인별 요소를 구분하게 하여, 개개인의 퍼포먼스를 인정해주고, 무임승차를 억제하기 Distinguish an individual component in the assessment of group work in order to acknowledge individual performance, as well as to discour- age free-riding 

7. 학생을 피드백과 평가에 참여시키기 Involve students in feedback and assessment 

•    provide clear guidelines and criteria (rubrics) at the start of the group work

•    apply self- and peer feedback/peer assessment

평가절차 Procedures

개인별 점수: 단순한 것부터 복잡한 것까지

The individual grade was most often based on an algorithm taking peer and/or self-assessment into account.

These procedures varied in complexity ranging from a holistic view (Lejk & Wyvill 2001, 2002) to a com- plex procedure—which

• normalized raw peer ratings,

• calcu- lated individual weighting factors,

• partially corrected for inter-rater agreement and

• constrained above-average con- tributions (Spatar et al. 2015).

 

대체로 Summative 한 특성이 있지만, 일부에서는 narrative feedback을 제공하게 하였고 이는 학생에게 유용하다.

Respondents to the survey reported a summative nature of group work assessment as the main purpose in all but one institution. of were Summative assessments group assignments reported to provide students with a qualification and also some kind of narrative feed-back (written or oral, provided standard or on request). Such narrative feedback may provide students with useful input for future learning. 

GW평가에서 무임승차자가 잠재적 문제였지만, 대부분은 이것을 크리티컬 하다고 보진 않았음.

Free riding is recognized as a potential problem in group work assessment by all of the seven medical schools that use the group work for summative assignments, but most do not regard it as a critical issue.

추가적 관찰결과

Additional findings from the questionnaire

GW는 교육적으로 협력을 배울 수 있는 수단으로 여겨지지만, 현실적인 이유도 있었다. MCQ나 다른 평가들은 더 노동집약적이다. GW평가에서는 supervision이나 correcting에 더 적은 교수만이 필요하다.

Although group work is seen as a means for learning to collaborate and thus is applied for educational reasons, it should be noted that respondents also explicitly mentioned practical reasons for applying group work. Compared to multiple-choice examinations, other forms of assessment, such as essays or papers, are more labor-intensive in terms of staff time needed for correcting. By using group assign- ments, fewer staff are needed for supervision and correcting compared to individual assessments.

고찰

Discussion

 

 

평가도구 Tools

조사 결과에 따르면 개인 기여에 대한 동료 평가는 네덜란드의 의과 대학의 정규 관행이다.

peer assessment of individual contributions. is in agreement with regular practice in medical schools in the Netherlands according to the findings based on the questionnaire.

우리의 검토에서 연구의 절반에서 자가 평가가 사용되었지만 Lejk and Wyvill (2002)과 Spatar et al. (2015)는 GW의 총괄평가에서 개개인의 (기여)요소를 식별하기 위해 자체 평가를 사용하지 말라는 권고한 바 있다. 자기 평가는 다양성을 감소시키고 (Lejk & Wyvill 2002), 무임승차자를 굳이 식별 할 필요가 없으며, 학생들은 종종 자신을 평가할 수없는 것처럼 보인다. 그러나 형성평가 목적이나 학습 기회를 위해서는 자체 평가가 여전히 중요 할 수 있습니다.

Although self-assessment is used in half of the studies in our review, we agree with Lejk and Wyvill (2002) and Spatar et al. (2015) who advise not to use self-assessment for identifying the individual component of group work in summative assessments.

• Self-assessment reduces the vari- ability (Lejk & Wyvill 2002),

• it is not necessary to identify free riders, and

• students often appear unable to assess themselves (Spatar et al. (2015) for an elaborate discus- sion on this issue).

Yet, for formative assessment and very learning opportunities, self-assessment can still be valuable.

평가준거 Criteria

개별 기여도 평가는...

The assessment of individual contribution may be derived from

• 산출물에 대한 개별 노력 the perceived effort individual students put in the group product and/or from

• 프로세스에 참여 the perceived participa- tion in the group process (e.g. attendance, active participa- tion, creativity).

Process와 Product의 구분에 관한 끊임없는 논쟁이 있다. 동료평가로는 process와 product를 구분할 수 없습니다. 결국 '기여도'라는 모호한 용어를 쓰게 된다. 동료 평가의 기준이 명확하지 않고 잘 정의되지 않은 경우 개인 기여도 평가는 인식된 노력perceived effort의 평가일 뿐입니다. 따라서 우리는 Process와 Product에 대한 학습 목표를 먼저 정의하고 이에 대한 명확한 기준을 수립하는 것이 중요하다는 점을 강조합니다 (Box Recommendation 7 참조).

A recurrent discussion in practice is the distinction between assessing the process or the product of the group work. With peer assessment, it is difficult to dif- ferentiate between process and product. This results in col- lating both with the vague term ‘‘contribution.’’ If the criteria for peer assessment are not clear and well defined, the assessment of individual contribution becomes only an assessment of perceived effort. Therefore, we stress the importance of first defining the learning goals on process and/or product and formulating clear criteria accordingly (see the Box recommendation 7).

평가절차 Procedures

거의 모든 연구에서 교사평가와 동료평가의 조합으로 개인 성적을 부여하였다. 동료 평가의 신뢰성에 대해 종종 의문이 제기된다 (Dancer & Dancer 1992, Stefani 1992, Pond et al 1995, Orsmond et al 1996, Falchikov & Goldfinch 2000). 또한 많은 연구자들이 최종 성적에 동료평가를 넣는 것에 대해 우려를 표한다.
 

In almost all studies, a combination of tutor and peer assessment was used to give an individual grade. The reli- ability of peer assessment is often questioned (Dancer & Dancer 1992; Stefani 1992; Pond et al. 1995; Orsmond et al. 1996; Falchikov & Goldfinch 2000) and various authors warn to be cautious in weighing peer assessment of contribution into the final grade.

 

이 두 가지를 비교하는 결정은 학습 목표에 기초해야합니다. 가장 중요한 학습 목표가 product에 의해서 다뤄진다면, 더 많은 가치가 추가되어야하지만, 팀 기술 또는 일하는 기술이 가장 중요하다면 동료 평가에 더 많은 무게를 줘야합니다. 이러한 결정은 언제나 일종의 타협이다.

• 최종 product에만 초점을 맞추는 것은 개개인의 기여를 고려하면 정의롭지 않다. 최종 product을 고려하지 않고 진공 상태에서 공동 작업 기술을 평가하는 것은 인위적입니다.

• 반면에, 팀의 공통 목표(product)가 채점 절차에서 중요하지 않게되면 팀의 기능에 영향을 미쳐 결과적으로 validity에도 영향을 준다.
  

The decision about weighing these two should be founded on the learn-ing objectives (the the Box recommendation 3). If the final product covers most important learning objectives,more value should be added to it, but if team skills or col-laboration skills are most important more weight should be given to peer assessment. Weighing different factors in the decision is always a compromise. Focusing purely on the end product will not do justice to individual contributions.Assessing collaboration skills in a vacuum without taking the final product into account is artificial. On the other hand, if the shared goal of the team (the final product)becomes unimportant in the grading procedure, it will influ-ence the functioning of the team and consequently the val-idity of the assessment of collaboration skills. 

GW평가에서 그룹 크기를 고려해야 한다. 

It is important to take the group size into account for group work assessment

보통 그룹 사이즈는 7명 이내로 작았고, 4~6명이 적절하다고 Storm은 말한다. 그룹 크기가 커짐에 따라 그룹의 점수에 개인 성과에 대한 정보가 적어 지므로, 개인 성과를 식별하는 것이 점차 중요해지게 된다. 따라서 팀이 클수록 individual component가 더 가중치를 받아야합니다.

The group sizes in the studies included in the review were small (maximum 7 students). According to Strom et al.(1999), four to six students per group is ideal. With increas-ing group size, a group mark becomes less informative of individual performance, so identifying individual perform-ance becomes increasingly important. Hence, the bigger the teams, the more weight the individual component should receive. 

팀이 유지되는 기간도 중요하다. 오랜 기간 지속되는 경우는 짧게 끝나는 경우와 다르다. GW동안의 개별기여에 대한 평가가 학생에게 중요한 피드백을 주기 때문에, 장기간 유지되는 팀에서는 다수의 저부담 형성평가가 좋다. 

Related to this is the duration of team composi- tions. A continuous group process over a longer period of time differs from a single end-of-course activity. Since evalu- ation of individual contributions during group work pro- vides students with valuable feedback, multiple formative low-stakes assessment moments over a longer period of time are preferred

학생들이 피드백을 받고 성찰해볼 수 있을 것 

This enables students to reflect upon the feedback received and improve their teamwork activities.

익명으로 진행될 수 있는데, 비밀이 보장될 때 솔직한 답변이 나온다. 익명일 때 spread of score가 더 높았다.

Finally, peer assessment can be done in the open or anonymously. When given in secret, more honest com- ments can be expected. Lejk and Wyvill (2001) found that the spread of scores is higher in anonymous peer assessment.

추가 이슈 Additional issues

 

(1)학생의 행동(태도), (2)그룹 구성 

During our screening and analysis of the literature, two add- itional issues in defining group work assessment emerged: (1) student behavior (or attitude) and (2) group compos- ition.

 

여러 연구에 따르면 일반적으로 학생들의 인식은 그룹에 긍정적인 방향으로 나타납니다 (예 : 기사 2004). 그러나 우리를 놀라게 한 점은 성적 시스템의 특성을 학생 행동에 연계시키는 연구가 없었기 때문입니다. Jin (2012)은 단지 공평성이 등급 시스템의 복잡성과 관련이 없다는 것을 발견했습니다. 학생들은 무임 승차 행동을 고려한 성적 시스템이 선호하지 않는 시스템보다 선호 함을 나타냅니다 (Maiden & Perry 2011). 다른 연구들은 또한 직원들과 학생들이 무임 승차 문제를 수입 주제로 간주하고 있음을 보여 주었다 (Maiden & Perry 2011; Spatar 외 2015). 그러나 무임승차자를 식별하는 것이 채점 시스템의 주요 목표가되어서는 안됩니다. 협동 기술에 대한 피드백을 제공하고 학생들의 강점과 약점을 파악하는 것이 더 중요합니다.

Multiple studies found that students’ perceptions work positive towards group are generally (e.g. Knight 2004). However, what struck us was that no study linked the characteristics of the grading system to student behav- ior. Only Jin (2012) found that perceived fairness was not related to the complexity of the grading system. Students do indicate that grading systems that take free-riding behavior into account are preferred over systems that do not (Maiden & Perry 2011). Other studies also indicate that staff and students regard the free-riding issue as an import- ant topic (Maiden & Perry 2011; Spatar et al. 2015). However, identifying free riders should not be the main goal of a grading system. Providing feedback on collabor- ation skills and identifying students’ strengths and weak- nesses should be more valuable.

 

두 번째 쟁점은 집단 구성 (Takeda & Homberg 2014, Dingel & Wei 2014, Spatar 외 2015)에 기인 한 편향에 관한 것이다. 우리는 그룹의 구성이 그룹의 기능에 영향을 줄 수 있음을 인정합니다. 피어 마킹에서 성별 편견에 대한 주장을지지하는 증거는 거의 없다 (Tucker, 2013). 그러나 평가 이전에 그룹 구성은 그룹 작업 중 공동 작업에 영향을 미칠 수있다. 예를 들어 여성의 팀 스킬이 더 높을 수 있고 (Strom et al. 1999), 성 균형 그룹이 불균형보다 공평한 기여를 하게된다는 증거가있다 그룹 (Takeda & Homberg 2014). 그러나 그룹 작업 평가를위한 그룹 구성의 실질적인 관련성은 코스의 그룹 구성이 종종 영향을 미치기 어렵 기 때문에 덜 명확합니다.

The second issue concerns biases due to group compos- ition (Takeda & Homberg 2014; Dingel & Wei 2014; Spatar et al. 2015). We acknowledge that the composition of the group is likely to influence how the group functions. There is little evidence to support an argument for gender bias in peer marking (Tucker 2013). However, prior to assessment, group composition may influence collaboration during group work—for example, women may have higher team- work skills (Strom et al. 1999) and there is evidence that gender balanced groups result in more equitable contribu- tions than imbalanced groups (Takeda & Homberg 2014). Still, the practical relevance of group composition for group work assessment is less obvious as the composition of groups in a course is often difficult to influence.

한계 Limitations

 

 

고찰

Conclusion

목표가 협업을 평가하는 것이라면 타당도는 단순히 product 이상이 되어야 하며, 그룹 전체 및 개인에 관한 process를 포함해야한다고 생각합니다.
if the goal is to assess collaboration, we believe the validity argument should also be based on more than a group product and should include the process, both regarding the group as a whole and its individuals.

문제는 여전히 남아 있습니다 : 그룹 작업 평가를위한 채점 시스템이 어떻게 설정되어야합니까? 다음을 고려하자

The question remains: how should a grading system for group work assessment be set up? In the Box, recommen- dations are provided, collected from the health faculties in the Netherlands. From the studies and the questionnaire, we conclude that the following steps should be considered when constructing and implementing group work assessment.

1. 주요 학습 목표는 무엇입니까? product and process 의 상대적 중요성에 대한 결정이 내려져야한다.
1. What are the main learning goals? A decision should be made about the relative importance of product and process.

2. 가중치 체계와 공식이 목적에 부합합니까? 동료 평가에 대한 기준이 잘 정의되어 있습니까? 동료 평가를 시작하기 전에 교사와 학생들 간의 동료 평가를 위한 그룹 작업 및 기준에 대한 공헌의 성격을 논의하는 것이 좋습니다.
2. Does the weighting scheme and formula fit the pur- pose? Are the criteria for peer assessment well defined? It is worth considering discussing the nature of the contributions to group work and criteria for peer assessment between tutors and students before start- ing the peer assessment.

3. 최종 product(과제)는 GW에 적합한가? (Box Recommendation 1 참조)
3. Is the end product (task) suitable for group work? (see Box recommendation 1)

4. 그룹 구성이 평가 결과에 bias을 의심 할만한 이유를 제시합니까? 만약 그렇다면  이를 방지하기 위해 어떤 안전 조치가 취해 있습니까?

4. Does the group composition give reason to suspect bias in assessment results? If yes: What safety measures are in place to counteract this?

5. 팀 스킬이 그룹에서 항상 분명하지는 않습니다. 이러한 기술을 개발할 수있는 지침과 기회를 제공하십시오. Box Recommendation 4. 마지막에뿐만 아니라 주기적으로 피드백을 제공하십시오.

5. Team skills are not always evident in groups. Provide guidance and opportunities to develop these skills— Box recommendation 4. Provide feedback periodically, not only at the end.

 

 

---

Med Teach. 2016 Jul;38(7):675-82. doi: 10.3109/0142159X.2016.1170796. Epub 2016 Apr 25.

Assessing the "I" in group work assessment: State of the art and recommendations for practice.

Dijkstra J1,2, Latijnhouwers M3,2, Norbart A4,2, Tio RA5,2.

Author information

• 1a Maastricht University, Maastricht University Office , Maastricht , the Netherlands ;
• 2e Special Interest Group on Assessment of the Netherlands Association for Medical Education (NVMO).
• 3b Radboud University Medical Center Nijmegen, Radboudumc Health Academy , Nijmegen , the Netherlands ;
• 4c Leiden University Medical Center , Leiden , the Netherlands ;
• 5d Department of Cardiology and Center for Educational Development and Research in Health Professions , University Medical Center Groningen , Groningen , the Netherlands.

Abstract

INTRODUCTION:

The use of group work assessment in medical education is becoming increasingly important to assess the competency of collaborator. However, debate continues on whether this does justice to individual development and assessment. This paper focuses on assessing the individual component within group work.

METHOD:

An integrative literature review was conducted and complemented with a survey among representatives of all medical schools in the Netherlands to investigate current practices.

RESULTS:

The 14 studies included in our review show that an individual component is mainly assessed by peer assessment of individual contributions. Process and product of group work were seldom used separately as criteria. The individual grade is most often based on a group grade and an algorithm to incorporate peer grades. The survey provides an overview of best practices and recommendations for implementing group work assessment.

DISCUSSION:

The main pitfall when using peer assessment for group work assessment lies in differentiating between the group workprocess and the resulting product of the group work. Hence, clear criteria are needed to avoid measuring only effort. Decisions about how to weigh assessment of the product and peer assessment of individual contribution should be carefully made and based on predetermined learning goals.

PMID:

 

27111741

 

DOI:

 

10.3109/0142159X.2016.1170796

[PubMed - in process]