보건의료전문직 교육훈련을 위한 학생선발에서의 MMI - Systematic review (Med Teach, 2013)

보건의료전문직 교육훈련을 위한 학생선발에서의 MMI - Systematic review (Med Teach, 2013)

The Multiple Mini-Interview (MMI) for student selection in health professions training – A systematic review

ALLAN PAU1, KAMALAN JEEVARATNAM2, YU SUI CHEN1, ABDOUL AZIZ FALL1, CHARMAINE KHOO1 &

VISHNA DEVI NADARAJAH1

1International Medical University, Malaysia, 2Royal College of Surgeons in Ireland, Perdana University, Malaysia

보건전문직 교육 프로그램 학생을 선발하는 것은 고부담 결정이다. 패널 혹은 위원회 면접이 흔히 사용되나 근거들을 살펴보면 이러한 방식은 학업 혹은 임상 수행능력 예측에 제한적 능력만 가진다.

Admissions to health professions training programmes are high stake decisions. The panel or board interview is commonly used to aid this decision (Edwards et al. 1990), although the evidence suggests its limited ability to predict academic or clinical performance in health care disciplines (Goho & Blackman 2006).

예를 들어 Dixon 등은 패널 인터뷰를 review하여 구조와 점수 anchor가 신뢰도와 타당도에 영향을 준다고 하였으며, Wilkinson 등은 패널 인터뷰가 예측력이 떨어지고 면접으로 인한 'threat'이 일부 잠재적 지원자를 떨어져나가게 한다고 하면서 GPA가 학업 수행능력에서 최고의 예측력을 가지는 것이라고 결론지었다.

For example, Dixon et al. (2002), in their review on the panel interview commented that structure and scoring anchors impact on its reliability and validity. Wilkinson et al. (2008), in their study argued that panel interviews have little predictive value and added that the ‘‘threat’’ of an interview may even dissuade some potential applicants and concluded that GPA (grade point average from student pre entry qualification) has the best predictive value to academic performance.

면접을 구조화하는 것은 수용가능도와 신뢰도를 향상시킨다. MMI는 고도로 구조화된 학생선발 방법이다.

Structuring the interview has been reported to enhance its acceptability and reliability (Patrick et al. 2001). The Multiple Mini-Interview (MMI) is a highly structured student selection method designed to resemble the Objective Structured Clinical Examination (OSCE) (Eva et al. 2004c).

MMI는 지원자의 역량에 대한 다면적 표집을 통해서 그들의 전체적 능력에 대한 더 구체적인 그림을 갖게 해준다.

The MMI, therefore, allows a wide sampling of candidates’ competencies in order to gain a more accurate picture of their overall ability.

방법

Methods

 

결과

Results

Review한 연구들의 특징

Characteristics of studies reviewed

MMI의 특징

Features of the MMI

• The number of stations used in the studies reviewed ranged from 4 to 12, with 10 studies using a 10-station MMI, 6 using 12, 5 using 8, and the remaining 9 using 4, 7, 9 or 11 stations.
• Fourteen of the studies used one assessor per station while 4 used 2 assessors, and the remaining 12 did not report the number of assessors per station.
• Most studies used faculty as assessors, while some used a combination of faculty and community practitioners (Hecker & Violato 2011) and others included students (Brownell et al. 2007).

• The range of time at each station was 5 to 15min with a mode of 8 min. Eleven studies reported using 8-min stations, five using 7-min, three using 10-min, one using 5-min and one 15-min stations.
• Two studies tested the effect of different lengths of time at stations; one comparing eight and six minutes (Cameron & Mackeigan 2012) and the other eight and five minutes (Dodson et al. 2009). Seven did not report the time at each station.

 

• The average MMI has 10 stations, each lasting eight minutes and is rated by one assessor.

 

 

활용가능성

Feasibility

Three studies reported on the feasibility of the MMI. One reported that it did not require more examiners when compared to the panel interview, did not cost more, and the interviews could be completed over a short period of time(Brownell et al. 2007; Finlayson & Townson 2011). Another study reported that it provided a positive experience for interviewers as well as applicants (Eva et al. 2004c).

수용가능성

Acceptability

Of the 30 studies reviewed, 14 reported on the acceptability of the MMI. Some authors reported that the MMI was acceptable to interviewees and interviewers because it was perceived as fair (Razack et al. 2009), transparent (Uijtdehaage et al. 2011) and providing opportunities for the interviewees to regain composure if they had problems with a previous station(Kumar et al 2009). Positive experience for both applicants and examiners has also been reported (Eva et al. 2004c). 

Acceptability was also determined as free from gender and cultural bias (Brownell et al. 2007), and socio-economic disadvantage (Uijtdehaage et al. 2011) or benefit of previous coaching (Griffin et al. 2008). Griffin et al. (2008) reported that previous coaching, as disclosed by applicants, had no effect on UMAT or MMI scores. Applicants who had previous MMI experience improved their subsequent performance in the same stations but not in new stations. 

Preference for station length differed between interviewers and interviewees, with the former judging six mins to be ‘‘just right’’ and eight mins to be ‘‘a bit long’’, and the latter preferring longer time (Cameron & Mackeigan 2012). One study reported that graduate candidates outperformed school-leavers (Dowell et al. 2012) while another reported no difference between graduate and school-leaver applicants(O’Brien et al. 2012). 

Acceptability of the MMI was compared to that of the panel or standard interview by O’Brien et al. (O’Brien et al. 2011) for graduate and school-leaver applicants to 4-year and 5-year medical training programmes. The 5-year candidates, generally school-leaver applicants, reportedly felt that the MMI gave amore accurate picture of their abilities and that the panel interview was more difficult. In contrast, the 4-year candidates felt the MMI was more difficult. 

신뢰도

Reliability

Eighteen studies reported on the reliability of the MMI. Intra-station reliability was reported to reach 0.98 by Lemay et al.(2007). The inter-item reliability (i.e. the internal consistency of the three scores assigned within any one station) and the inter-rater reliability within stations have also been reported to be very high by Dore et al. (2010). However, Finlayson &Townson (2011) conducted a 4-station MMI, each at 15min,and reported inter-rater reliability ranging from 0.50 to 0.69 for three stations, and 0.10 for one station. 

Generally the reported reliability ranged from moderate(Roberts et al. 2008) to acceptable (Dore et al. 2010) to high(Lemay et al. 2007), with Cronbach’s alpha ranging from 0.69to 0.98. However, Finlayson & Townson (2011) reported 0.45 inter-station reliability ranging from to 0.47. Other researchers have also reported low inter-station correlations,(Lemay et al. 2007). 

Using generalisability analysis, Hecker & Violato (2011) reported a G coefficient of 0.79 for seven stations with two assessors. A Decision study indicated that G¼0.81 can be achieved fromten stations with one assessor. Similarly, in Dore et al.’s (2010) study, G¼0.55 to 0.72 for seven stations, is increased to G¼0.64 to 0.79 with 10 stations in a D-study.

타당도

Validity

내용 타당도

Content validity.

The validity of the MMI was discussed in 17 of the 30 studies. One key observation was that the MMI scores did not correlate with traditional admission tools scores such as (r ¼0.185), the personal interview undergraduate grades (r ¼0.317), simulated tutorial (r ¼ 0.227) and autobiograph- (r ¼0.170) ical sketch (Eva et al. 2004c). Other studies did not reported that the MMI correlate with pre-entry such as academic scores (Hecker qualifications, the GPA et al. 2009), pre-pharmacy average (PPA) (r ¼ 0.025) or (r ¼0.042) Pharmacy College Admission Test (PCAT) (Cameron & Mackeigan 2012), GAMSAT ( ¼0.04) and UK Clinical Aptitude Test (MCAT) ( ¼ 0.00) (O’Brien et al. 2011).

 

However, positive association with certain cognitive skills, such as the GAMSAT scores for ‘‘Reasoning in (r ¼0.26) Humanities and Social Sciences’’ and ‘‘Written Communication’’ (0.26) (Roberts et al. 2008), and cognitive reasoning skills (Roberts et al. 2009) have been reported as well as correlation with autobiographical submission focusing on ethical decision making (r ¼0.65) (Dore et al. 2006). The MMI was not reported to be associated with emotional intelligence (Yen et al. 2011).

예측 타당도

Predictive validity.

For medical students, MMI performance at admission was the best predictor for subsequent OSCE as well as clerkship performance (Eva et al. 2004a). Validity against future non-cognitive assessment was investigated by Eva et al. (2009), who reported that MMI performance at admission was statistically significantly predictive of perform- ance at future examinations, such as the percentage of stations passed in the MCCQE (Medical Council of Canada Qualifying Examination) Part II.

 

However, a cross-sectional study investigating the association between MMI performance of medical residency applicants and their MCCEE (Medical Council of Canada Evaluating Examination) and MCCQE I scores reported low, non-significant correlations, and also non-significant correlation with MCCQE II scores (Hofmeister et al. 2009). In a more recent study, Eva et al. (2012) reported that better MMI performance at entry to medical school was predictive of higher MCCQE scores.

Discussion

이번 연구의 핵심 결과는 다음과 같다.

The key findings were that the MMI was

• (i) practically feasible in terms of efficient utilisation of time, costs and human resources when compared to the panel interview;
• (ii) generally acceptable to both interviewees and interviewers;
• (iii) generally reliable with acceptable Cronbach’s apha and G-coefficient values; and
• (iv) predictive of future performance in certain aspects of medical council examinations.

스테이션을 개발하고 면접을 시행하려면 전문성이 필요하다. 따라서 초기의 준비비용은 높을 수 있다.

Expertise is also necessary in developing the stations and conducting the interviews. Therefore the initial preparatory costs to develop the MMI are likely to be high (Rosenfeld et al. 2008).

Kumar 등은 시나리오 기반의 MMI가 어떻게 답변해야 하는가에 대한 리허설이나 코칭을 더 어렵게 만들며, 실제로도 MMI에서의 수행능력이 자기-보고된 이전 코칭 여부와 상관이 없으며, 코칭을 받지 못한 지원자에게 불리하지 않음을 보여주었다.

Kumar et al. (2009) identified that the scenario-based nature of the MMI made it harder for rehearsal and coaching of responses, and indeed it has been reported that performance at the MMI is not associated with self-reported previous therefore, coaching (Griffin et al. 2008), and does not disadvantage applicants with no access to coaching.

스테이션 내, 평가자 간 신뢰도는 높고 스테이션 간 신뢰도는 낮은데, 이는 서로 다른 스테이션은 서로 다른 특질을 테스트하기 때문이다.

For example, it is expected that intra-station and inter-rater reliability would be high and inter- station reliability low(Lemay et al. 2007; Dore et al. 2010) since different stations may test different attributes.

그러나 신뢰도는 스테이션이나 면접관의 수와 관련된 것으로 보이며, 각 스테이션의 내용과도 관련되어 있다.

However, reliability would appear to be associated with number of stations or interviewers (Hecker & Violato 2011), and the content of each station (Lemay et al. 2007).

MMI 신뢰도는 acceptable하며, Ottawa 2010 컨퍼런스에서 보건의료전문직 선발에서 활용되는 것에 대한 합의를 이루었다. 면접관의 주관은 측정오차의 가장 큰 원인이 되며, 면접관 훈련이 도움이 될 것임을 시사한다.

The reliability of the MMI has generally been reported to be acceptable. This has been recognised by the Ottawa 2010 Conference in a consensus statement on assessment for al. selection for the health care professions (Prideaux et 2011). Interviewer subjectivity is the largest source of meas- urement error, suggesting that interviewer training could be helpful (Roberts et al. 2008).

대부분의 연구는 MMI 수행능력이 입학 전 성취(GPA, MCAT, GAMSAT)과 무관함을 보여준다. 이는 MMI가 비인지적 특성을 평가한다는 것이다.

Most studies reported that MMI performance was not pre-entry qualifications such as associated with academic GPA, MCAT and GAMSAT scores. This suggests that the MMI is capable of testing non-cognitive attributes, such as

• profes- sionalism (Hofmeister et al. 2009),
• legal, ethical and organ- isational skills. (Eva et al. 2009),
• motivation, interest in medicine, decision making skills, ability to debate a complex issue (O’Brien et al. 2011),
• empathy, moral and ethical reasoning, motivation and preparedness to study medicine, teamwork and leadership, honesty and integrity (Till et al. 2013), and
• advocacy, ambiguity, collegiality and collabor- ation, cultural sensitivity, responsibility and reliability (Lemay et al. 2007).
  

Lemay JF, Lockyer JM, Collin VT, Brownell AK. 2007. Assessment of non- cognitive traits through the admissions multiple mini-interview. Med Educ 41(6):573–579.

---

Med Teach. 2013 Dec;35(12):1027-41. doi: 10.3109/0142159X.2013.829912. Epub 2013 Sep 20.

The Multiple Mini-Interview (MMI) for student selection in health professions training - a systematic review.

Pau A1, Jeevaratnam K, Chen YS, Fall AA, Khoo C, Nadarajah VD.

Author information

• 1International Medical University , Malaysia.

Abstract

BACKGROUND:

The Multiple Mini-Interview (MMI) has been used increasingly for selection of students to health professions programmes.

OBJECTIVES:

This paper reports on the evidence base for the feasibility, acceptability, reliability and validity of the MMI.

DATA SOURCES:

CINAHL and MEDLINE STUDY ELIGIBILITY CRITERIA: All studies testing the MMI on applicants to health professions training.

STUDY APPRAISAL AND SYNTHESIS METHODS:

Each paper was appraised by two reviewers. Narrative summary findings on feasibility, acceptability, reliability and validity are presented.

RESULTS:

Of the 64 citations identified, 30 were selected for review. The modal MMI consisted of 10 stations, each lasting eight minutes and assessed by one interviewer. The MMI was feasible, i.e. did not require more examiners, did not cost more, and interviews were completed over a short period of time. It was acceptable, i.e. fair, transparent, free from gender, cultural and socio-economic bias, and did not favour applicants with previous coaching. Its reliability was reported to be moderate to high, with Cronbach's alpha = 0.69-0.98 and G = 0.55-0.72. MMI scores did not correlate to traditional admission tools scores, were not associated with pre-entry academic qualifications, were the best predictor for OSCE performance and statistically predictive of subsequent performance at medical council examinations.

CONCLUSIONS:

The MMI is reliable, acceptable and feasible. The evidence base for its validity against future medical council exams is growing with reports from longitudinal investigations. However, further research is needed for its acceptability in different cultural context and validity against future clinical behaviours.

PMID:

 

24050709

 

[PubMed - indexed for MEDLINE]