의과/치의과 대학원 선발에 있어서 인터넷 기반 MMI의 활용

의과/치의과 학생 선발에 있어서 인터넷 기반 MMI의 활용

Internet-based multiple mini-interviews for candidate selection for graduate entry programmes

David Tiller, Deborah O’Mara, Imogene Rothnie, Stewart Dunn, Lily Lee & Chris Roberts

배경

2006년 이래로 The University of Sydney graduate medical and dental programmes에서는 Multiple Mini Interviews (MMI)를 활용해왔다. 2011년에는 국제 지원자(international candidates)를 대상으로 Skype를 이용한 (internet based MMI, iMMI)를 시행하였으며, 국내 지원자는 직접 MMI를 수행하였다. 우리는 이 두 가지 방법의 MMI에서 나온 점수가 서로 동등한지를 살펴보았다. 또한 iMMI의 실현가능성, 비용-효과성, 수용가능성 등을 살펴보고자 하였다.

방법

2011 국제 지원자의 iMMI결과를 2009 국제 지원자의 MMI점수 및 2011년의 국내 지원자 MMI점수와 비교하였다. ANOVA를 사용하여 두 가지 형식의 MMI를 비교하였다. 이 과정의 수용가능성(acceptability)에 대한 정보는 피면접자와 면접자의 피드백을 통해서 얻었으며, 비용 절감(cost savings)를 추산하였다.

결과

2011 iMMI점수와 2009년의 MMI점수에는 유의미한 차이가 없었다. 또한 국내 지원자와 국제 지원자간 2011년에 점수의 차이는 없었다. 국제 지원자의 MMI점수는 variation이 더 컸다. 일반화가능성 이론(generalizability theory)를 이용했을 때 9개의 iMMI 문항의 신뢰도는 0.76이었고, MMI는 0.70이었다. iMMI 상황의 전달은 원활히 이루어졌으며 면접자와 면접관은 형식과 전달에 대한 피드백을 주었다. 비용절감은 50,000호주달러였으며, 84%의 절감효과가 있었다.

결론

우리는 이 연구가 High stake interview에 있어서 최초의 인터넷 기반 MMI에 대한 보고라고 생각한다. 여기서 면접관들은 iMMI를 통해서 타당하고 신뢰성 있는 결정을 내릴 수 있음을 보여주었고, 그 과정은 참여자도 수용할 수 있었고, 비용은 절감하면서도 직접 하는 MMI와 동등한 결과를 보여줬다. iMMI에서 분산이 조금 더 큰 것에 대한 향후 연구가 필요할 것이다.

---

Equivalence of MMI and iMMI scores

Two separate one-way analyses of variance (ANOVAs) were used to investigate the equivalence of the two formats by exploring whether the medium of interviewing resulted in significantly different mean scores for the in-person MMI and the internet-based iMMI. 

In the first analysis, the iMMI scores for international candidates in 2011 were compared with the in-person MMI scores for international candidates in 2009. To ensure that differences in mean scores were not attributable to the different application process for the two time periods and differences in the MMI questions, a comparison was also made for both cohorts for 2011. Thus, the scores for the iMMI for international candidates in 2011 were compared with the scores for the in-person MMI for local candidates in 2011. In accordance with the assumptions for ANOVA, the homogeneity of variance between each comparison was tested according to Levene’s statistic. Where the two groups were shown to have unequal variance, the non-parametric Brown–Forsythe median test was used to assess the difference between the average scores. Eta squared was used to examine the proportion of variance associated with the main effects assessed in the ANOVA.

The equality of the two formats was also investigated through a variance components analysis using a minimum norm quadratic unbiased estimation in order to compare effects for the 2011 iMMI candidates and the in-person MMI. We assumed that all effects were random given the unbalanced design. We estimated the variance due to ‘candidate’, ‘question’, ‘interviewer’ and ‘question X interviewer’. Interviewers varied in the number of circuits they participated in, ranging from one to 15 for both formats, with an average of four circuits for the iMMI and three for the MMI. However, the interaction of ‘candidate X interviewer’ and ‘candidate X question’ could not be separated from each other because interviewers were confounded with MMI questions within each candidate and, thus, remained in the error term. There were insufficient candidates per circuit of individual questions to include ‘circuit’ as a variable in the analysis.

Reliability of the iMMI

A generalisability coefficient was calculated to provide the estimate of reliability for the iMMI and the MMI. The formula to determine the generalisability coefficient has been published elsewhere.4

(4 Roberts C, Walton M, Rothnie I, Crossley J, Lyon P, Kumar K, Tiller D. Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. Med Educ 2008;42 (4):396–404.)

---

Med Educ. 2013 Aug;47(8):801-10. doi: 10.1111/medu.12224.

Internet-based multiple mini-interviews for candidate selection for graduate entry programmes.

Tiller D, O'Mara D, Rothnie I, Dunn S, Lee L, Roberts C.

Source

The University of Sydney, Sydney, Australia.

Abstract

CONTEXT:

Multiple mini-interviews (MMIs) have been used by The University of Sydney graduate medical and dental programmes since 2006. In 2011, interviews with international candidates were conducted using Skype (iMMI), whereas interviews with local candidates were conducted in person. We determined whether the MMI scores derived from both methods were comparable. We describe the feasibility, acceptability and cost-effectiveness of the iMMI.

METHODS:

We compared 2011 international student internet-based iMMI results with data from 2009 international student MMIs and 2011 local student MMIs. Analyses of variance (anovas) were used to investigate equivalence of the two formats by exploring whether the medium of interviewing resulted in significantly different mean scores and variance for the in-person MMI and the iMMI. Acceptability of the process was informed by feedback surveys from interviewers and candidates, and cost savings were estimated.

RESULTS:

No significant difference was found between the 2011 iMMI scores for international candidates and MMI scores in 2009 (p > 0.05). There was no significant difference between the MMI scores for local and international candidates in 2011 (p > 0.05); the MMI scores for international candidates had greater variation (p < 0.01). Using generalisability theory, the reliability of the nine-question iMMI was 0.76 and for the MMI was 0.70. Delivery of the iMMI occurred smoothly and candidates and interviewers gave positive feedback on its format and delivery. Cost savings have been estimated to be over AU$50 000, representing an 84% saving.

CONCLUSIONS:

We believe this is the first study reporting an internet-based MMI for a high stakes interview. We have shown that interviewers were able to make valid and reliable decisions about candidates through the iMMI in a process that was acceptable to participants, producing comparable results to the in-person MMI with a saving of resources. The slightly wider variance in iMMI scores warrants further investigation.

© 2013 John Wiley & Sons Ltd.

PMID:

 

23837426

 

[PubMed - in process]