어떤 요인(개인적/외부적)이 학생들의 학습목표 생성에 영향을 줄 것인가? (Acad Med, 2010)

어떤 요인(개인적/외부적)이 학생들의 학습목표 생성에 영향을 줄 것인가? (Acad Med, 2010)

Which Factors, Personal or External, Most Influence Students’ Generation of Learning Goals?

Kevin W. Eva, Juan Munoz, Mark D. Hanson, Allyn Walsh, and Jacqueline Wakefield

자신의 개인적 강점, 약점 및 학습 필요성에 대한 판단을 내리는 데 있어 자기자신이 다른 사람보다 나은 위치에 있다는 것에 대한 논쟁은 쉽다.

It is easy to make an argument, that the individual is better positioned than anyone else to make judgments about his or her personal strengths, weaknesses, and learning needs.

그러나 유감스럽게도 개인이 끌어 내야하는 다양한 정보원이, 자신의 능력에 대한 정확한 인상을 유발하는 방식으로 mentally aggregated되지 않는 경우가 많다는 논쟁도 쉽게 제기됩니다 .2 "자신의 스킬 수준에 대해서 개인적으로 생성한 요약 판단"으로 정의되는 자기 평가는 명백히 능력을 판단하기에 부족함이 있다. 이것은 우리 자신은 특정한 순간에 특정한 문제에 관해 성공할 가능성을 판단 할 수 있다는 증거가 있음에도 불구하고 그러하다 3,4 이러한 문헌을 근거로 많은 사람들은 수행능력 개선에 도움을 줄 외부 피드백이 필수적이라고 주장했다. 그러한 피드백은 시험이라든가, practice에 대한 자료, 동료나 감독관에 의해서 제공된 guidance등이 있다.

Unfortunately, however, it has become equally easy to argue that the various sources of information on which one has to draw are often not mentally aggregated in a manner that engenders an accurate impression of one’s abilities.2 Self- assessment, defined as an individually generated summary judgment of one’s skill level, is clearly insufficient for judging ability, despite findings that suggest we can judge our likelihood of success with a specific problemat a specific moment in time.3,4 This literature has prompted many to argue that external feedback is essential to help guide performance improvements, be that feedback delivered in the formof testing, data pertaining to one’s practice, or guidance provided by a colleague/supervisor.5

그러나 여기에 흥미로운 역설이 등장하기 시작했다. 개인적인 경험은 정확한 자기 평가의 생성에 충분하지 않다. 따라서 더 나은 피드백을 찾아나서야call for한다. 그러나 이러한 피드백은 이미 결함이 있는 자기 평가의 맥락에서 전달deliver되고 수용received되고 만다. 즉, 피드백은 결코 진공 상태에서 이뤄지지 않으며, 자기자신에 대해서 갖고 있는 인상의 신뢰도와 무관하게, 항상 그 인상의 맥락에서 해석 될 것이다. 사실, 연구는 MSF의 가치를 어떻게 인식하느냐는 그 피드백이 피드백을 받는 사람의 자기평가와 얼마나 쉽게 조화 될 수 있느냐에 달려있다고 제안했다.

Herein, however, an interesting paradox is beginning to emerge. Personal experiences do not appear sufficient for the creation of accurate self-assessments; this leads to calls for better feedback— yet, such feedback must be delivered and received within the context of the flawed self-assessments. That is, feedback never occurs in a vacuum and, as such, will always be interpreted in the context of one’s impressions of one’s own ability, regardless of the trustworthiness of those impressions. Indeed, research has suggested that the perceived value of multisource feedback depends on the ease with which the feedback can be reconciled with one’s self-assessment.6

Method

Setting

Students in McMaster University’s undergraduate MD program completed a 10-station preclerkship OSCE 15 months into their medical training. This was the second OSCE in which these students had participated, their first having been sat nine months earlier. The OSCE took place in a single evening with half the class beginning the assessment at 6 PM and the other half beginning the assessment at 9:00 PM.

Procedure

Data collection took place during two rest stations, built into the OSCE circuit at equal intervals, and after completion of the OSCE. During the rest stations participants were asked to complete a data sheet for each of the one to five stations they had encountered prior to entering the rest station (the number being dependent on where in the circuit the student began). 

평가자는 다음의 좋은 피드백 기준을 얼마나 따랐는가?

On each sheet, students were asked, “To what extent do you agree that the evaluator modeled each of the following components of good feedback?” The individual components were drawn from the feedback literature and a prior study run in a similar manner.9 They were 

• (1) 변화 될 수있는 행동의 특정 관찰에 대한 피드백 기반,

• (2) 긍정적이고 부정적인 피드백의 균형을 건설적인 방식으로 사용,

• (3) 피드백을 (속도를 조절하여) 몇 가지 유용한 점으로 제한

• (4) 미래 학습 목표에 피드백을 연결하는 데 학생을 참여

• (1) Based feedback on specific observations of behavior that can be changed, 

• (2) Used a balance of positive and negative feedback in a constructive manner, 

• (3) Paced feedback, limiting it to a few usable points, and 

• (4) Engaged the student in linking feedback to future learning objectives. 

In each case, a seven-point Likert scale ranging from Strongly Disagree to Strongly Agree was included. Participants were then asked, 

• (1) 이 평가자에게 받은 피드백에 몇 점을 주겠는가? 
  “Overall, how would you rate the quality of feedback received from this examiner?” with an accompanying seven- point scale ranging from Poor to Excellent, 

• (2) 평가자가 몇 점을 주었는가?
  “What score did the evaluator assign to your performance on this station?” and 

• (3) 스스로 생각하기에 얼마나 잘한거 같은가?
  “How well do you think you performed on this station?” 

Ten-point scales accompanied the latter two questions, each of which was anchored in the same way as the scale provided to the examiners.

After completing the OSCE, students were brought into another room and asked to complete the above questions for the one to five stations they had encountered since their second rest station. At that point they were provided with a second data collection form on which they were asked to write down 

• (1) any learning goals they had generated during the OSCE, and 

• (2) what activities they would engage in to fulfill those goals. 

For each generated response, students were asked to indicate the station number that corresponded to the learning goal/activity if they “became aware of this need as a result of performance on a particular station.”

Analysis

Two regression analyses were conducted to determine which variables predicted 

• (1) the generation of learning goals and 

• (2) stated learning activities in which students intended to engage. 

The dependent variable for each analysis was whether or not a learning goal/activity was generated in response to a particular station because it was anticipated that the factors considered would be variable across station. Dummy variables were created for both station and participant to isolate the variance attributable to these factors. 

Additional independent variables included in the analysis were 

• (1) the order in which the station was encountered,

• (2) the student’s opinion of his or her performance, 

• (3) the student’s opinion of the faculty observer’s feedback, and 

• (4) the faculty observer’s opinion of the student’s performance. 

The latter variable was operationalized as the faculty rater’s opinion as reported by the student because the student’s memory of the rater’s opinion was thought to be more important for the research question than the actual rating. The squared multiple correlation between each variable and all others was used to test for multicollinearity, and Tabachnik and Fidell’s10 stringent criterion of tolerance  0.10 was used to rule out variables whose coefficients were artificially inflated because of multicollinearity.

결과

Results

At completion of the OSCE, students generated a total of 88 learning goals (mean  2.2 per person). In predicting whether or not students generated a learning goal in response to their experience with a particular station, a regression analysis was used as described above. 

The model accounted for 45.0%of the variance, with 

• student (P  .001), 

• the order in which stations were encountered (later stations were more likely to prompt learning goals than were earlier stations; P  .002), and 

• the student’s self- evaluation (P  .001) statistically predicting whether or not a learning goal was generated.

Similarly, students identified a total of 68 activities in which they intended to participate to satisfy the above-stated learning goals (mean  1.7 per student). In predicting whether or not students identified a learning activity in response to their experience with a particular station, a regression analysis was used as described above. The model accounted for 49.9%of the variance, with 

• student (P  .001), 

• the order in which stations were encountered (P  .02), and 

• the student’s self-evaluation (P  .001) statistically predicting whether or not a learning goal was generated.

고찰

Discussion

이러한 결과는 자기 평가가 스스로 능력을 나타낼 수는 없지만 학생의 의견이 학습 목표의 생성에 지배적으로 영향을 미치는 방식을 결정하는 것은 여전히 ​​중요하다는 증거를 제공합니다. 이 연구의 맥락에서, 학생들은 [시뮬레이션 된 환자와 상호 작용]하고, [시험관과의 실적을 논의 할 기회]를 얻은 후에 자기 평가를 하였다. 따라서 이러한 판단은 Boud11에서 정의한 "다른 출처로부터 정보를 얻어 내린 자기 평가"에 부합한다.

These findings provide evidence that, while self-assessments may not themselves validly indicate ability, it is still critical to determine how students perceive their ability as their opinion may dominantly influence the generation of learning goals. In the context of this study, self-assessments were generated after the students interacted with the simulated patient and had the opportunity to discuss their performance with their examiners. As such, these judgments constitute self-assessments as defined by Boud11 in that they are self- assessments that are informed by other sources.

그러나 학생들은 자신이 얼마나 잘 수행했는지에 대한 심사관의 의견을 단순히 받아들이지 않았습니다.

however, students did not simply adopt the opinions of their examiners regarding how well they had performed

시험관 평점과 학생 평점 사이의 상관 관계는 완벽하지 못했으며, 스스로의 시험점수에 대한 학생의 평가는 시험관 평가 만 사용한 것보다 학습 목표 생성에 대한 더 나은 예측을 제공했습니다.

the correlation between the examiner ratings and the student ratings was not perfect and the students’ opinions of the ratings they deserved provided better prediction of the generation of learning goals than did the examiner assessment alone.

피드백의 품질이 미치는 영향이 크다는 사실은(비록 marginal하게만 그렇더라도), [자기평가]와 [실제 수행능력]이 서로 misalign된 상태에서 어떤 것이 '좋은' 피드백인가에 대해 생각해볼 점이 많음을 시사한다. 그러나 Shute12가 발표 한 광범위한 문헌 검토에는 제공되는 피드백에 동의하지 않을 때 학생들이 어떻게 반응하는지에 대한 시험은 포함되어 있지 않습니다.

The finding that perceived quality of feedback was influential (though only marginally so) leads us to believe that there is much to be gained from better understanding what constitutes good (i.e., productive) feedback in situations where self-judgments and performance are misaligned. An extensive literature review published by Shute12 , however, has not included examinations of how students respond when they disagree with the feedback provided.

높은 피드백 품질 등급이 부여 되었음에도 불구하고, 스테이션 특정 피드백은 학생들의 자각 변화와 관련하여 거의 효과가 없었다. 즉, 자기 평가는 여러 스테이션에 걸쳐서 안정적이었다.

despite the high quality-of-feedback ratings assigned, the station-specific feedback did little with respect to altering students’ self-perceptions. That is, the self- assessments were relatively stable across station,

자기 방어와 낙관주의가 일반적인 적응적 경향이라고 주장하는 사람들이 예측하는 바와 같이 우리의 자기 인식은 피드백에 의해 쉽게 영향을받지 않는다 .15 이러한 견해는 흔히 말하는 피드백이 맥락 특이적이어야 한다는 개념을 지지한다. 왜냐하면, 그러한 피드백은 자기-인식에 대한 모욕을 주지 않기 때문이다.

our self-perceptions are not easily influenced by feedback as would be predicted by those who argue that ego defense and optimism are generally adaptive tendencies.15 This view supports the commonly espoused idea that feedback should be specific to the context because such feedback does not entail as great an affront to one’s self-perceptions.

[자기 평가는 능력에 대한 올바른 징후를 제공하지 못하며], 교육자가 학생의 성과 향상을 돕기 위해서는 자기평가가 오류에 빠져있기 쉽다는 것을 고려해야 한다고 보는 것이 안전할 것이다.

In the meantime, however, we think it safest to conclude that while self-assessments may not provide valid indications of ability, fallible self-assessments must be taken into account by educators hoping to help students improve their performance.

---

Acad Med. 2010 Oct;85(10 Suppl):S102-5. doi: 10.1097/ACM.0b013e3181ed42f2.

Which factors, personal or external, most influence students' generation of learning goals?

Eva KW1, Munoz J, Hanson MD, Walsh A, Wakefield J.

Author information

1

University of British Columbia, Vancouver, BC, Canada. kevin.eva@ubc.ca

Abstract

BACKGROUND:

While concern has been expressed about the validity of self-assessments, external feedback is likely filtered through self-assessment. This paper explores the relationship between self-assessments and feedback uptake.

METHOD:

During an objective structured clinical examination, students were asked to evaluate their performance and rate the quality of feedback provided by observers. Afterward, they were asked to list learning goals they generated, to indicate what activities they would undertake to fulfill those goals, and to identify which station(s) led them to generate each response. Regression analyses were used to determine which variables predicted the generation of goals/activities.

RESULTS:

Students' perceptions of their own performance were more likely to result in the generation of goals/strategies than was observer feedback or student perceptions of observer feedback quality. Later stations were more likely to result in goal/strategy generation than earlier stations.

CONCLUSIONS:

While self-assessments may not validly indicate ability, it is still critical to determine how students perceive their ability because their opinions drive their learning goals.

PMID:

 

20881690

 

DOI:

 

10.1097/ACM.0b013e3181ed42f2