의과대학 선발에서 MMI를 통한 비인지적 역량 측정(Med Educ, 2007)

의과대학 선발에서 MMI를 통한 비인지적 역량 측정(Med Educ, 2007)

Assessment of non-cognitive traits through the admissions multiple mini-interview

Jean-Franc¸ ois Lemay,1 Jocelyn M Lockyer,2 V Terri Collin3 & A Keith W Brownell4

입학면접은 학생선발과정 중 가장 주관적이면서 다양한 측면을 지닌 단계이다. 일반적으로 입학면접은 비인지적 특성을 평가하는 가치가 있는데, 그 신뢰도에 대한 의문이 있어왔다. 이러한 이유로 Michael G DeGroote School of Medicine at McMaster University 에서는 MMI를 도입했다.

The interview has been identified as among the most subjective and variable aspects of the medical school admissions process.1–4 Generally, the admissions interview is valued for its ability to assess non-cognitive attributes, although the reliability of the interview process has been questioned.5,6 In response to this challenge, the Michael G DeGroote School of Medicine at McMaster University intro- duced a multiple-station-based assessment process, the multiple mini-interview (MMI).6–8

MMI는 전통적 면접보다 더 신뢰도가 높으며, 임상실습 수행능력을 더 잘 예측해준다. 또한 의과대학이 중요시하는 가치를 반영할 수 있다. MMI는 모든 지원자들이, 성별에, 면접시간에, 배경에 비뚤리지 않는 결과를 제공한다.

MMI is more reliable than traditional interviews,6–8 is better able to predict pre-clerkship performance7 and can be designed to reflect the values of the medical school.9 Furthermore, the MMI appears to offer an unbiased opportunity for all applicants, regardless of gender, time of day, or background.6,8,10

일부 학생들에게 면접 내용을 미리 제공한 경우에도 MMI는 두 집단 사이에 차이를 보이지 않았다.

An assessment of the MMI when some candidates were provided with test content a priori and others were not showed no differences between the 2 groups.11

2003년 우리 의과대학은 선발인재상을 정립했다.

In 2003, our Faculty of Medicine adopted a list of non-cognitive attributes that we wanted our medical students to possess.12

Michael G DeGroote School of Medicine의 MMI가 4 가지 인성측면에만 초점을 뒀지만 우리는 더 다양한 측면을 평가하고자 했다.

Although the Michael G DeGroote School of Medicine’s MMI had only focused on 4 attributes (ethics, critical thinking, communication skills and knowledge of the health care system8,9), we felt the MMI could be developed to assess multiple non-cognitive attributes.

Each of the first 9 MMI stations was designed to assess a distinct non-cognitive attribute: 

1 advocacy; 

2 ambiguity; 

3 collegiality and collaboration; 

4 cultural sensitivity; 

5 empathy; 

6 ethics; 

7 honesty and integrity; 

8 responsibility and reliability, and 

9 self-assessment. 9

마지막인 10번째 스테이션에서, 지원자들은 스스로 왜 우수한 의과대학생과 의사가 될 것인지를 답하게 했다.

At the 10th and last station, applicants were asked why they thought they would become an excellent medical student and doctor.

각 스테이션의 템플릿은 비슷했다. 5~15줄의 시나리오, 2분간 읽고, "discuss this with the assessor"로 끝남.

The template for each station was similar. The applicant was provided with a 5–15-line scenario to read during a 2-minute period prior to entering the room. Each scenario ended with the statement: Discuss this with the assessor. 

방에 들어가서 8분간 진행. 평가자들은 지원자가 말하게 놔두고, 중간에 코멘트를 하거나 개입하지 않도록 지시를 받았음. 또한 4개의 탐색질문을 하게 되어있음. 평가자는 각 스테이션의 목적과 배경정보를 제공받았음. (Appendix) 면접 2주 전 모든 평가자는 MMI에 대한 오리엔테이션과 2시간의 트레이닝 세션을 마쳤고, MMI의 개요와 연구 결과, 동일한 스테이션에서 2명의 지원자의 영상, 평가 시스템에 대한 설명을 들었다.

After entering the room, applicants discussed the scenario with the assessor for 8 minutes. Interview assessors were instructed to allow the applicant to talk at length and not to interrupt the commentary. The interview assessors were provided with 4 probing questions to use if they deemed it necessary. Assessors were provided with information about the objectives of the station and some background information. An example of a complete station is provided in the Appendix. Two weeks prior to the MMI, all assessors were oriented to the MMI in a 2-hour training session in which we provided an overview of the MMI and its research basis, showed a videoclip with 2 different applicants interviewed on the same station, and explained the scoring system.

각 스테이션에서 10점짜리 5개의 준거로 평가하였음. 

At each station, each applicant was assessed on 5 criteria using a 10-point scale. Thus applicants could achieve a score of up to 50 points for each station. The criteria were: 

• ability to understand and address the objectives of the scenario, 
• communication skills dis- played, 
• strength of the arguments presented, 
• suitability for a career in medicine, and 
• overall performance.

평가자들에게 지원자는 이미 충분한 자격을 갖추었으나 지원자간 차이를 둬야 하며, 처음 3~4명의 지원자를 통해서 calibration을 할 것을 권했다. 연습 세션에서 1점에서 10점까지 고루 사용할 것을 권했다.

We reminded interview assessors that all the applicants were highly qualified but that they would have to discriminate between applicants. We suggested inter- view assessors use the first 3 or 4 applicants to calibrate their scoring for the station. In the training session, we particularly emphasised to the interviewers that they needed to use the full range of the 10-point scales.

각 지원자는 9명의 평가자를 만났으며, 2일간 진행되었고, confidentiality를 유지하기 위해서 2개의 내용과 맥락이 비슷하지만 동일하지는 않은 parallel station을 만들었다. 둘 사이에 유의미한 차이는 없었다.

Thus, each applicant was assessed by 9 different interview assessors. As interviews were conducted on 2 days, to ensure confidentiality, 2 parallel stations with similar, but not identical, content and context were created for each non-cognitive attribute. As an ANOVA com- paring station performance for each day did not reveal any significant differences, we combined the data for days 1 and 2 for this study.

통계분석

For each applicant we had the following data: scores on each of the 10 stations (5 scales) as well as the total score, sociodemographic data (age, gender, grade point average [GPA]) and whether the applicant was accepted or placed on the waitlist. Descriptive statistics for the applicant pool for both the sociodemographic data and each station were tabulated. The internal consistency reliability (Cronbach’s alpha) was examined for each station (Question 1). 

To determine whether the stations were measuring a single construct or several, we correlated the total scores for each station. As a result of the multicolinearity among the subscale scores (within each station), the total scores for individual stations were used in the analysis. Additionally, an exploratory factor analysis (EFA) was used to determine whether the structure of the data was unidimensional or multidimensional. For the EFA, we analysed the data using principal component analysis, with varimax rotation following the Kaiser rule (i.e. eigenvalues >1.0), to determine the number of factors to be extracted (Question 2). 

We assessed the ability of the stations to discriminate between those placed on the acceptance and waiting lists by examining the mean scores for each station through an ANOVA (Question 3). 

ANOVA was used to determine whether there were sociodemographic differences (gender, GPA, age, degrees) between those accepted and those waitlisted (Question 4).

Cronbach's alpha는 item cohesiveness를 보여준다.

The high Cronbach’s alpha scores suggest there was high item cohesiveness among the subscales with each station and provide evidence of stable scores for each applicant

흥미롭게도 10번 스테이션을 포함하자는 결정은 현재 의과대학 학생과 교수들이 지원자들도 스스로에 대해 이야기할 기회를 줘야 한다고 주장하여 포함되었다. 지원자들은 미리 이런 스테이션이 있을 것이라는 사실을 전달받았다. 그러나 이 자료를 부면 평가자의 신념이 개인적인 특성을 가진다는 것을 확인시켜주었으며, 입학위원회가 MMI를 가지고 특정 인적특성을 평가해야지, 방향이 없는 토론이 되게 놔두면 안된다는 것을 보여준다. 10번째 스테이션의 이러한 결과는 평가자가 평가기준을 calibration할 기회가 없었기 때문일수도 있다. 이러한 유형의 스테이션에서는 얻을 것이 많지 않아 보이며, borderline에 있는 지원자에게는 영향을 미칠 수도 있다.

Interestingly, the decision to in- clude this station was made at the urging of our current medical students and faculty members, whofelt that applicants should have an opportunity to talk about themselves and present their cases for being accepted into our medical school. Applicantswere advised in advance of the interview day that this would be among the stations. However, these data reveal the idiosyncratic nature of assessor beliefs and provide a compelling reason for admissions committees to use the MMI to assess specific attributes and not to have a free-flowing discussion. The data on this 10th station were probably also compromised by the fact that asses- sors did not have an opportunity to calibrate applicants and may have been influenced by the prior performance of the candidate. There appears to be little to be gained from this type of station and it could affect admission for borderline candidates.

상관관계 분석으로부터 스테이션간 상관관계가 낮았음을 보여준다. 그러나 요인분석은 10개의 서로 다른 요인을 보여줌으로써, content specificity와 더불어 다면적 평가를 했음을 입증해주었다. 

Our correlation analyses showed a low correlation between stations; however, the factor analysis revealed 10 distinct factors, attesting to the multi- dimensional structure of the data as well as providing support for the existence of content specificity. This gives us confidence that the MMI is able to assess and differentiate between a large and diverse set of traits.

11 Reiter HI, Salvatori P, Rosenfeld J, Trinh K, Eva KW. The effect of defined violations of test security on admissions outcomes using multiple mini-interviews. Med Educ 2006;40 (1):36–42.

12 University of Calgary. Non-cognitive qualities we look for in students admitted to the Faculty of Medicine. University of Calgary. http://admissions.myweb.med. ucalgary.ca/NoncognitiveQualitiesWeLookFor.html. [Accessed 18 September 2006.]

---