3학년 학생의 임상수행능력평가와 학생과 평가자 성별의 관계(Acad Med, 2017)

3학년 학생의 임상수행능력평가와 학생과 평가자 성별의 관계(Acad Med, 2017)

Clinical Performance Evaluations of Third-Year Medical Students and Association With Student and Evaluator Gender
Alison Riese, MD, MPH, Leah Rappaport, MD, Brian Alverson, MD, Sangshin Park, DVM, MPH, PhD, and Randal M. Rockney, MD 

 

 

 

의대 졸업생을 레지던트 과정으로 선발하는 것은 여러 가지 요인에 의해 추진된다. 그러나 프로그램 책임자에 따르면 가장 중요한 선발 기준은 필수 핵심 임상실습의 성적이다.1 대부분의 핵심 임상 임상실습에서 의대생을 위한 평가 및 채점 도구로 임상 성과 평가(CPE)가 사용된다. 의대생들과 함께 일하는 임상의들은 지식 및 전문직업성은 물론, 병력 청취, 사례 발표 등 각 학생의 기본적인 임상 능력에 대한 공식적인 평가를 완료해야 한다. 대부분의 임상실습에서 이러한 평가는 표준화된 서면 검사 및 객관적인 구조화 임상 검사(OSCE)와 함께 학생들의 최종 임상실습 성적이 결정되는 데이터를 제공합니다. 
Selection of graduating medical students into residency programs is driven by multiple factors. However, according to program directors, the most important selection criteria are students’ grades on required core clerkships.1 Clinical performance evaluations (CPEs) are used in most core clinical clerkships as assessment and grading tools for medical students. Clinicians who work with medical students are asked to complete formal evaluations of each student’s basic clinical skills, such as history taking and case presentation, as well as fund of knowledge and professionalism. In most clerkships, these evaluations, along with standardized written examinations and objective structured clinical examinations (OSCEs), provide the data from which students’ final clerkship grades are determined. 

의대에 입학하는 과정과 임상실습 이전학년에 걸쳐 발생하는 수많은 평가는 대체로 표준화되었으며, 평가자-의존적 편향을 보이지 않을 것 같다. 반면 임상실습 성과에 대해서는 보다 주관적인 방식으로 평가된다. 
The numerous evaluations that occur over the course of attaining entrance to medical school and during the preclinical years are largely standardized and unlikely to exhibit grader-dependent bias. In contrast, medical students are evaluated in a more subjective manner when being assessed on their clinical performance. 

교육 분야의 문헌에 따르면, 학생의 성별은 종종 학생들을 대하고 점수를 매기는 데 있어 중요한 역할을 한다.5,6

• 초등학교에서는 비슷한 시험 점수에도 불구하고 여학생들이 남학생들보다 더 좋은 점수를 받게 되는데, 일부 연구자들은 이러한 점수를 "비인지적 기술"로 보고 있다. 구체적으로 말하자면, "배움에 대한 더 발전된 태도"입니다.
• 또한, 교사의 성별은 교육 역량과 성과에 대한 [기대와 인식]에 영향을 미칠 수 있습니다.
• 나아가, 연구들은 성별 짝짓기가 "역할 모델 효과"를 통해 학생들의 참여와 행동을 향상시킬 수 있다고 제안한다. 반대로 성별 불일치는 "고정관념 위협"을 유발할 수 있다. 부정적인 고정관념을 확인시켜줄 것 같은 불안감이 성능 저하로 이어질 수 있다.

Literature from the education field has shown that student gender often plays a role in how students are treated and graded.5,6

• In primary schools, girls are awarded better grades than boys, despite similar test scores, which some researchers attribute to “noncognitive skills”—specifically, “a more developed attitude towards learning.”6
• Additionally, teachers’ gender can affect their expectations and perceptions of educational competence and performance.7,8
• Furthermore, studies9–11 suggest that gender pairing can enhance, through a “role-model effect,” student engagement and behavior, or, conversely, gender noncongruence may induce “stereotype threat,” in which anxiety that one will confirm a negative stereotype can lead to a decrement in performance. 

일부 소규모 연구는 표준화된 환자(SP)에 의한 의대생들의 OSCE에 대한 임상 성과 시뮬레이션의 점수에서 학생과 평가자 성별 간의 상호작용을 제안했다. 
A few small studies12–14 have suggested an interaction between student and evaluator gender in the grading of medical students’ simulated clinical performance on OSCEs by standardized patients (SPs). 

학생과 평가자의 성별에 대한 유사한 점수 차이가 nonsimulated 임상 환경에 대한 몇 가지 소규모 연구에서 발견되었다.

• 작은 연구결과, 남자 프리셉터가 여학생들에게 가장 높은 평균 점수를 매긴 것으로 나타났다. 그리고 가장 낮은 평균 점수는 여자 프리셉터가 남학생에게 준 점수였다.
• 내과 레지던트 평가 연구에서는, 남성 레지던트는 (여성 주치의보다) 남성 주치의에게서 더 높은 점수를 받았다.
• 반대로 산부인과 의대생 평가에 대한 연구 결과 여학생들이 필기시험과 OSCE에서 더 나은 성적을 보였지만, 교수에 의한 평가는 남학생들이 받은 점수와 비슷했다.

Similar disparities in grading regarding student and evaluator gender have been found in a few small studies of nonsimulated clinical settings.15,16

• A small study showed that the highest mean grade was given by male preceptors to female students, and the lowest mean grade was given by female preceptors to male students.
• In a study of evaluations of internal medicine residents, male residents received higher grades from male attendings than from female attendings.17
• Conversely, a study of medical student grading in obstetrics– gynecology18 found that female students performed better on written exams and OSCEs; however, they were graded similarly to male students by their faculty evaluators. 

임상등급 과제의 객관성을 높이기 위한 모든 노력의 첫 단계로, 평가자의 학생 임상성적 채점에 어떤 문제가 영향을 미치는지 충분히 이해할 필요가 있다. 

As a first step in any effort to increase objectivity in clinical grade assignment, it is necessary to fully understand what issues influence evaluators’ grading of student clinical performance. 

방법
Method

이것은 Alpert 의과대학(AMS)에서 시행된 후향적 연구였다. 2013-2014학년도 동안 3학년 핵심 임상실습에서 의과대학 채점 데이터베이스(OASIS)에 기록된 CPE 4,462개가 처음에 포함되었다. AMS에서 핵심 임상실습과 연구 기간 동안의 기간은 내과(12주)와 수술, 산부인과, 가정의학과, 소아과, 정신의학과(매 6주)로 구성되었다.  
This was a retrospective study conducted at the Alpert Medical School (AMS). All 4,462 CPEs recorded in the medical school’s grading database (OASIS) from third-year core clerkships during the 2013–2014 academic year were initially included. At AMS, the core clerkships and their duration during the study period consisted of internal medicine (12 weeks) and surgery, obstetrics– gynecology, family medicine, pediatrics, and psychiatry (each 6 weeks). 


CPE 이수시점에 각 평가자가 선택할 수 있는 성적은 '특수', '기대 이상', '기대 미달', '기대 미달'이었다. 중복 항목이거나 일차 결과 또는 예측 변수에 대한 데이터가 불완전할 경우 평가는 제외되었습니다. 또한, "기대 미만" 등급의 CPE는 이 등급의 드문 발생(<1%) 때문에 제외되었다. 
The possible grades that could be selected by each evaluator completing a CPE were “exceptional,” “above expectations,” “meets expectations,” and “below expectations.” An evaluation was excluded if it was noted to be a duplicate entry or if data were incomplete for the primary outcome or predictor variables. Additionally, CPEs with a grade of “below expectations” were excluded because of the rare occurrence (< 1%) of this grade. 

우리는 식별되지 않은 데이터를 제공받았기 때문에 객관적인 비임상 평가와 그 데이터를 일치시킬 수 없었다. 그러나 우리는 2015년 수업의 미국 의료면허시험(USMLE) 1단계 점수를 남성과 여성의 점수로 비교했다. 
Because we were provided deidentified data, we were not able to match those data with any objective nonclinical evaluations. However, we did compare the United States Medical Licensing Examination (USMLE) Step 1 scores for men versus women in the class of 2015. 

모든 통계 분석은 SAS 9.4(SAS Institute, Carry, North Carolina)를 사용하여 수행되었다. P 값 ..05는 통계적으로 유의한 것으로 간주되었다. 이 연구에서는 카이-제곱 검정을 사용하여 최종 등급과 성별 및 공변량과의 연관성을 조사했습니다. 계층적 순서형 회귀 모형화는 학생과 평가자 특성이 학생의 성적에 미치는 영향("예외", "기대 이상", "기대 충족")을 조사하기 위해 수행되었으며, 둘 이상의 학생을 평가한 평가자의 비독립성 또는 "클러스터링"에 대해 조정되었습니다. 

All statistical analyses were performed using SAS 9.4 (SAS Institute, Cary, North Carolina). A P value < .05 was considered to be statistically significant. This study examined the associations of final grade with gender and covariates using chi-square tests. Hierarchical ordinal regression modeling was conducted to examine the effects of student and evaluator characteristics on a student’s grade (“exceptional,” “above expectations,” or “meets expectations”), adjusting for nonindependence, or “clustering,” of evaluators who rated more than one student.

일변량 모형에서 P 값이 .05인 성별 및 공변량은 단계적 선택 절차에 의해 구축된 다중 변수 회귀 모형에 통합되었습니다. 잔차 분산을 유의하게 감소시킨 변수는 최종 모형에서 유지되었습니다. 
Gender and covariates with a P value < .05 in the univariable model were incorporated into a multivariable regression model, which was built by the stepwise selection procedure. Variables that significantly reduced residual variance were retained in the final model.

동시성을 피하기 위해 두 독립 변수에 대해 파이 계수를 추정했습니다. 변수 간의 높은 공리성이 관측된 경우(r > 0.6) 다중 변수 모형화에 대한 학생의 성적과 가장 관련된 변수를 선택했습니다. 가정의학과 정신의학 분야에서의 평가 횟수가 적기 때문에, 다변수 모델링을 위해 이러한 전문 분야의 데이터가 결합되었습니다. 주효과 모형을 만든 후 유의성에 대한 교호작용 항이 탐색되었습니다. 
To avoid colinearity, phi coefficients were estimated for two independent variables. If high colinearity among variables was observed (r > 0.6), we selected the most relevant variable to the student’s grade for multivariable modeling. Because of the small number of evaluations in family medicine and psychiatry, data from these specialties were combined for the multivariable modeling. After the main effects model was built, interaction terms were explored for significance. 

결과
Results

따라서 최종 연구 데이터 세트는 4,272개의 CPE로 구성되었으며, 이는 155명의 학생의 성과와 관련하여 829명의 평가자가 완료하였다. 평균(SD) USMLE 단계(학생 및 평가자 통계는 표 1 참조)
Thus, the final study dataset comprised 4,272 CPEs, which were completed by 829 evaluators regarding the performance of 155 students. The mean (SD) USMLE Step (See Table 1 for student and evaluator demographics.)

학생당 CPE 수(중간값 27, IQR 6–39)와 평가자당 CPE 수(중간값 3, IQR 1–7)에 변동이 있었다. 각 임상실습, 학생 및 평가자 특성은 받은 성적 분포에서 통계적으로 유의한 차이와 관련이 있었다. (표 2 참조) 
There was variability in the number of CPEs per student (median 27, IQR 6–39) and CPEs per evaluator (median 3, IQR 1–7). Each clerkship, student, and evaluator characteristic examined was associated with a statistically significant difference in the distribution of grades received. (See Table 2.) 

단일 변수 모형에서는 모든 예측 변수가 등급과 연관되었습니다. 교수 연령과 교육 수준(파이 계수 0.84) 간의 상관관계가 높기 때문에 다변수 모형에는 평가자 연령만 고려되었습니다. 점수 변동의 총 32.9%는 다중 변수 모델에서 within-evaluator nesting of grades(내포 등급 상관 계수 = 0.329, P < 0.001)에 의해 설명되었다. 일변량 모형의 모든 유의한 차이는 다변량 모형에서 유지되었습니다. 다중 변수 모형에서 여학생 성별은 더 높은 성적과 연관되었습니다(수정 오즈비[AOR], 1.30; 95% CI, 1.13–1.50). 여성 교수진의 성별은 낮은 성적과 관련이 있었다(AOR, 0.72, 95% CI, 0.55–0.93). 관찰 시간이 길어진 것, 학생 연령이 높은 것, 평가자 연령이 낮은 것 모두 높은 성적과 관련이 있었다. 내과 평가자는 더 나은 점수를 줄 확률이 가장 높았고 산부인과 평가자는 가장 낮았다. (표 3 참조) 
In univariable models, all predictors were associated with the grade. Because of high correlation between faculty age and training level (phi coefficient 0.84), only evaluator age was considered for the multivariable model. A total of 32.9% of the variability in the grades was accounted for by within-evaluator nesting of grades in the multivariable model (intraclass correlation coefficient = 0.329; P < .001). All significant differences in the univariable models were retained in the multivariable model. In the multivariable model, female student gender was associated with higher grades (adjusted odds ratio [AOR], 1.30; 95% CI, 1.13–1.50). Female faculty gender was associated with lower grades (AOR, 0.72; 95% CI, 0.55–0.93). Longer observation time, older student age, and younger evaluator age were all associated with higher grades. Evaluators in internal medicine had the highest odds of giving a better grade, while those in obstetrics–gynecology had the lowest odds. (See Table 3.) 

다른 모든 주효과에 대해 보정했을 때, [학생과 교수 성별 간의 교호작용]도 유의했습니다(P = 0.03; 그림 1 참조). 남성 평가자는 남학생과 여학생의 점수(P = .29)에서 큰 차이는 없었지만, 여성 평가자는 여학생에 비해 남학생에게 낮은 점수를 주었다(P <.001).
The interaction between student and faculty gender, adjusted for all other main effects, was also significant (P = .03; see Figure 1). Male evaluators did not significantly differ in their grading of male and female students (P = .29); however, female evaluators gave lower grades to male students compared with female students (P < .001). 

또한, [교수 연령과 교수 성별 간에 유의한 교호작용]이 발견되었으며(P = .047), 나이 든 남성 평가자는 젊은 남성(P = 0.001)에 비해 현저히 낮은 점수를 주었지만(P = .71) 여성 연령대의 성적 차이는 유의하지 않았다. (그림 2 참조). 학생 성별과 학생 연령 사이에는 교호작용이 없었습니다(P = .63).  
Additionally, a significant interaction between faculty age and faculty gender was found (P = .047), with older male evaluators giving significantly lower grades than younger men (P = .001), while there was no significant difference in grading for the female age groups (P = .71). (See Figure 2). There was no interaction between student gender and student age (P = .63). 

 

 

고찰
Discussion

그러나, 우리의 연구 결과는 의대생과 의대생 사이의 임상실습 성적등급의 불일치discrepancy가 주로 여성 평가자에 의해 발생했음을 보여준다.
However, our findings show that the discrepancy in clinical performance grades between male and female medical students was driven primarily by female evaluators. 

의대생 임상성과에 대한 남녀 평가자의 평가 차이가 가장 복잡하다. 의대생들의 임상 성과는 의학적 지식과 임상 통찰력 이외의 속성에 의해 영향을 받는다. 실제로 두 연구에서 공감을 보인 의대생들이 임상평가에서 더 좋은 평가를 받았고, 여성이 남성보다 공감 척도에서 더 높은 점수를 받았다고 보고했다.  
The discrepancy between male and female evaluators’ assessment of medical students’ clinical performance is most perplexing. Medical students’ clinical performance is influenced by attributes outside of medical knowledge and clinical acumen. Indeed, two studies22,23 reported that medical students who showed empathy received better clinical evaluations, and women scored higher on empathy scales than men did. 

임상환경에서 여성이 남성을 능가한다는 문헌의 본문을 적용한다면, 

• 여성 평가자는 여학생에서 우수한 성적을 정확하게 검출한 반면 
• 남성 평가자는 이러한 차이를 감지하지 못했거나 채점방식에 치우쳤다는 것을 알 수 있다. 

If the body of literature showing that women outperform men in the clinical setting is applied, our findings suggest

• that female evaluators accurately detected superior performance in their female students,
• while male evaluators either were unable to detect these differences or were biased in their grading methods. 

그러나 이번 연구 결과는 성별과 학업 성취도, 평가 간의 훨씬 더 복잡한 상호작용을 부각시킬 가능성이 높다. 초등교육계와 마찬가지로 여학생의 '학습 태도'도 한몫할 수 있고, 동성 평가자의 역할 모델 가능성과 이성 평가생의 고정관념적 위협도 있어 평가자의 성별에 따라 학생에게 영향을 미칠 수 있다. 또 다른 문제가 될 수 있는 것은 환자가 학생의 성별에 따라 의대생과 다르게 상호작용할 수 있다는 점이며, 이로 인해 의대생들의 성과 평가에도 영향을 미칠 수 있다는 점이다.
However, it is likely that this finding highlights an even more complicated interplay between gender and academic performance and assessment. As in the primary education world, female students’ “learning attitude” may also play a role, as well as the possible role modeling of same-gender evaluators and the stereotype threat of opposite-gender graders, which may influence students to perform differently depending on the gender of their evaluators. Another potential complicating matter is that patients may interact differently with medical students depending on the student’s gender, which could also affect the assessment of their performance.

원인이 무엇이든 간에, 우리의 연구결과는 남녀 학생들이 각기 다른 임상성과를 경험하고 있으며, 평가자의 성별이 이러한 차이를 일으키는 독립적인 동인임을 시사하고 있다는 점에서 우려된다.
Whatever the cause, it is concerning that our study findings suggest that male and female students experience different gradings of their clinical performances, and that the gender of the evaluator is an independent driver of this difference. 

우리의 데이터는 또한 [평가자 연령과 성별 간에 유의한 상호작용]을 발견했으며, 젊은 남성 평가자가 모든 연령 그룹에서 나이 든 남성 평가자보다 높은 점수를 수여했다. 젊은 평가자들이 다른 연구에서 더 관대한 학년인 것으로 밝혀진 반면, 우리가 아는 바로는 연령-성별 상호작용은 다른 연구에서는 조사되지 않았기에, 이러한 발견은 추가적인 조사를 필요로 한다. 다시 한 번, 내적 평가자의 특성이 학생들의 차별성 평가를 초래했다는 점을 우려한다. 평가자들에 대한 훈련이 필요하거나, [평가자가 공정한 임상실습 점수를 줄 능력이 있는지]를 고려할 때 [평가자의 특성]을 고려해야 한다.
Our data also found a significant interaction between evaluator age and gender, with younger male evaluators awarding higher grades than older male evaluators and than female evaluators in all age groups. While younger evaluators have been found to be more lenient graders in other studies,27,28 to our knowledge the age–gender interaction has not been examined elsewhere, and this finding warrants additional investigation. Again, it is concerning that intrinsic evaluator characteristics have led to differential grading of students. Either improved training of graders is needed, or the characteristics of the evaluators must be taken into account when considering their ability to give fair clerkship grades. 

우리의 자료는 또한 우리 학교의 임상실습에서 [전공과목별로 평가등급에 상당한 차이]를 보여주는데, 이 결과는 다른 많은 학교에도 적용될 것이다. CPE에 대한 일관된 접근법을 제공하기 위해 이러한 가변성을 검사해야 한다. CPE를 살펴볼 때 학생들이 평가자와 함께 보내는 시간뿐만 아니라 서로 다른 핵심 임상실습의 구조와 기간의 차이를 고려해야 한다. 
Our data also demonstrate substantial differences in the way clerkship students are graded by department at our school, a finding that we suspect applies to many schools. This variability should be examined to provide a consistent approach to CPEs. Differences in the structure and duration of the different core clerkships, as well as the time students spend with evaluators, must be taken into consideration when looking at CPEs.

데이터 집합에서 개별 수준 데이터를 사용할 수 없기 때문에 임상 성과 등급을 표준화된 테스트 점수와 조정하거나 비교할 수 없었습니다. 또한, 우리는 성별 표현과 2013-2014년 의과대학에서의 성별 상호작용이 성별 관계와 세대 차이가 다른 방식으로 데이터를 왜곡할 수 있는 예년과 매우 다를 수 있다는 것을 인정한다.

We were not able to adjust for or compare clinical performance grades with standardized test scores, since the individual-level data were not available in our dataset. Further, we recognize that gender representation, and thus gender interactions at a medical school in 2013–2014, might be very different from what was obtained in previous years, when gender relationships and generational differences would perhaps skew data in other ways. 

 

---

Acad Med. 2017 Jun;92(6):835-840.

 doi: 10.1097/ACM.0000000000001565.

Clinical Performance Evaluations of Third-Year Medical Students and Association With Student and Evaluator Gender

Alison Riese 1, Leah Rappaport, Brian Alverson, Sangshin Park, Randal M Rockney

Affiliations collapse

Affiliation

• 1A. Riese is assistant professor, Department of Pediatrics and Medical Science, Section of Medical Education, Alpert Medical School of Brown University, Providence, Rhode Island.L. Rappaport is a first-year pediatrics resident, University of Michigan Medical School, Ann Arbor, Michigan.B. Alverson is associate professor, Department of Pediatrics and Medical Science, Section of Medical Education, Alpert Medical School of Brown University, Providence, Rhode Island.S. Park is postdoctoral research associate, Alpert Medical School of Brown University and Center for International Health Research at Rhode Island Hospital, Providence, Rhode Island.R.M. Rockney is professor, Department of Pediatrics, Family Medicine, and Medical Science, Section of Medical Education, Alpert Medical School of Brown University, Providence, Rhode Island.

• PMID: 28099178
• DOI: 10.1097/ACM.0000000000001565Abstract
• Purpose: Clinical performance evaluations are major components of medical school clerkship grades. But are they sufficiently objective? This study aimed to determine whether student and evaluator gender is associated with assessment of overall clinical performance.Results: Female students were more likely to receive a better grade than males (adjusted odds ratio [AOR] 1.30, 95% confidence interval [CI] 1.13-1.50), and female evaluators awarded lower grades than males (AOR 0.72, 95% CI 0.55-0.93), adjusting for department, observation time, and student and evaluator age. The interaction between student and evaluator gender was significant (P = .03), with female evaluators assigning higher grades to female students, while male evaluators' grading did not differ by student gender. Students who spent a short time with evaluators were also more likely to get a lower grade.
• Conclusions: A one-year examination of all third-year clerkship clinical performance evaluations at a single institution revealed that male and female evaluators rated male and female students differently, even when accounting for other measured variables.
• Method: This was a retrospective analysis of 4,272 core clerkship clinical performance evaluations by 829 evaluators of 155 third-year students, within the Alpert Medical School grading database for the 2013-2014 academic year. Overall clinical performance, assessed on a three-point scale (meets expectations, above expectations, exceptional), was extracted from each evaluation, as well as evaluator gender, age, training level, department, student gender and age, and length of observation time. Hierarchical ordinal regression modeling was conducted to account for clustering of evaluations.