던디(Dundee) 대학에서 수행한 Multiple mini-interview(MMI)의 예측타당도(Predictive validity) 연구

(출처 : http://www.nytimes.com/2011/07/18/opinion/l18docs.html?_r=0)

Introduction

의과대학의 학생 선발에 있어서 학업적 성취도와 더불어 비인지적(non-cognitive), 또는 비학업적(non-academic)한 특성이 중요하다는 것은 잘 알려져 있다. 2004년 맥마스터 대학에서 개발된 이후, Dundee 대학은 multiple mini interview (MMI)를 주요한 전입학(pre-admission) 평가법으로 사용해왔다.

It is widely accepted that so-called ‘non-cognitive’ or ‘non-academic’ attributes (such as interpersonal skills and moral reasoning) are important for medical school selection in addition to academic achievement. 1 Developed and introduced at McMaster in 2004, Dundee has since adopted the multiple miniinterview (MMI) as the primary pre-admissions measure

for this purpose. Other schools in the UK are increasingly following suit.

MMI는 OSCE와 같은 형태로 여러 방에서 순차적으로 면접을 보면서, 지원자를 다양한 측면에서 바라보면서(multiple snapshots) 다양한 인성적 특성을 평가하기 위한 목적으로 쓰인다. 이러한 종류의 면접은 Eva 등에 의해서 psychometric 특성의 측정을 위해서 처음 개발되었다.

MMIs aim to assess a broad array of candidates’ personal characteristics through ratings from multiple snapshots of behaviour in an objective structured clinical examination (OSCE)-like rotational approach. This type of interview was first introduced by Eva et al.2 because of the need for an interview process with robust psychometric properties, unlike most traditional interviews.

많은 내용으로 많은 면접자들을 평가한 결과 MMI는 지원자의 행동특성에 대한 정확한 그림을 얻는데 효과적임이 밝혀졌다. 미국, 호주, 영국에서 많은 연구결과가 있었고 MMI는 점차 전세계적으로 확산중에 있다.

By testing a larger content sample with multiple independent interviewers, MMIs have demonstrated that they can offer a more accurate picture of a candidate’s behaviour.3 With compelling evidence on reliability and other satisfactory psychometric properties from the USA, Australia and the UK,2,4–8 MMIs continue to be adopted across medical and dental schools worldwide.

이제 관심은 MMI가 의과대학과 의과대학 졸업 이후에까지의 예측력을 가지는가에 대한 것에 쏠리고 있다. 많은 연구를 통해 MMI의 결과가 미래의 수행능력과 통계적으로 유의미하면서 실용적인 관계를 보인다는 것이 확인되었다.

Attention has now shifted to the ability of MMIs to predict performance in medical school and beyond. A number of studies have demonstrated that they show statistically significant and practically relevant relationships with future performance.9–11

비록 이러한 연구들이 높은 예측타당도를 보여줬지만, 대부분의 연구가 소수의 캐나다 중심의 코호트로 수행되었다는 한계가 있다.

Although these studies have successfully demonstrated predictive validity, it is clear that more research is needed as the majority of this work was based on the same small Canadian cohort

따라서 북아메리카 이외의 지역에서, 그리고 더 많은 수의 코호트에서 MMI의 예측타당도를 검증해볼 필요가 있다.

Therefore, the body of evidence examining the predictive validity of MMIs would benefit from an analysis of different and larger cohorts and from outside of North America

MMI를 활용하는 것의 효용성은 이미 확실해졌지만, 다른 입학 기준에 대해서도 같은 기대를 가질 수 있다. 따라서 MMI가 다른 기준들보다 더 예측력이 뛰어난가는 확인해 볼 필요가 있다.

Although it is certainly beneficial to consider the usefulness of MMIs, the same expectation should be set for all admissions measures.9 It is therefore important to consider the predictive ability of MMIs relative to other pre-admissions measures

Ferguson 등과 Siu와 Reiter는 의과대학에서의 성공 예측에 있어서 자기소개서는 아무런 예측력이 없음을 밝혔다. Wright와 Bradley는 자기소개서를 평가한 점수가 의과대학에서의 수행능력을 예측하지 못하는 것은 물론이고, 오히려 사회경제적 배경에  따라 특정 지원자에게 더 이득을 주는 식의 비뚤림(bias)만 가지고 있다고 밝혔다.

Ferguson et al.13 and Siu and Reiter14 reviewed predictors of success in medical school and found that there was a lack of evidence that personal statements or references have any predictive value in subsequent achievement. Wright and Bradley15 also found that not only did scores derived from the personal statement fail to predict medical school examination performance, but they were also biased towards those from more advantaged socio-economic backgrounds.

UKCAT은 지식 검사로서 의과대학과 치과대학에서 필요한 다양한 범위의 정신적 능력(mental abilities)를 평가하기 위한 시험이다.

The UKCAT (http://www.ukcat.ac.uk) is an intelligence test used to ‘assesses a range of mental abilities identified by university medical and dental schools as important’.16

여러 문헌에서 MCAT, GAMSAT, BMAT이 모두 일부 미래의 수행능력을 예측할 수 있음이 밝혀졌지만, UKCAT에서 같은 종류의 연구결과가 나온 적은 없다.

Though the literature suggests that the MCAT, GAMSAT and BMAT each have some success at predicting future performance, level of success has not been replicated so far with the UKCAT

Statistical Analysis

Correlations were adjusted for range restriction and are referred to in this study as ‘unrestricted’ correlations. Statistical significance was determined prior to correcting the correlations. This adjustment is common in predictive validity studies and is carried out to counter correlation underestimates when the observed sample is not representative of the population of interest.21,22

The strengths of correlations were compared using Cohen’s effect size interpretations (small 0.10, medium 0.30, large 0.50)24 and the US Department of Labour, Employment Training and Administration’s guidelines for interpreting correlation coefficients in predictive validity studies (‘unlikely to be useful’ < 0.11; ‘dependent on circumstances’, 0.11–0.20; ‘likely to be useful’ 0.21– 0.35; ‘very beneficial’ > 0.35).25

Discussion

이 연구에서는 두 개의 서로 다른 코호트에서 MMI가 의과대학 시험의 성취도와 일관된 예측력을 가진다는 것을 보여줌으로써 MMI의 타당도에 대한 중요한 근거를 제시하고 있다.

this study does provide important evidence of the validity of the MMIs by demonstrating that it was the most consistent predictor of success in medical school examinations across two separate cohorts and years

비록 상관관계의 크기가 그다지 높지는 않지만, 이정도 수준의 예측타당도만으로도 지원자의 수가 많고 정교한 선발 결정이 중요한 선발시스템에서는 상당한 가치를 지닐 수 있음이 주장된 바 있다. 범위제한(range restriction)을 한 뒤에도, 이들 계수(coefficient)들은 '도움이 되어 보임' 또는 '상당한 도움이 됨' 이라고 나왔다. 상관관계는 OSCE평가에서 가장 높았으며, 이는 아마도 '의사소통 능력'이나, '압박적인 상황에서 수행하는 능력'과 같은 공통적 요소를 평가했기 때문이라고 생각된다.

Although the size of these correlations can be described as moderate, it has been asserted that measures with even modest predictive validity could add considerable value to selection systems where the ratio of applicants to places is large and the importance of sound selection decisions is high.28 After adjusting for range restriction, these coefficients can be described as ‘likely to be useful’ or ‘very beneficial’.25 Correlations were largest in OSCE assessments, perhaps because certain components are common in both, such as communication skills, or even more generally an ability to ‘perform under pressure’.

---

Med Educ. 2013 Jul;47(7):717-25. doi: 10.1111/medu.12193.

Predictive validity of the Dundee multiple mini-interview.

Husbands A, Dowell J.

Source

Division of Clinical and Population Sciences and Education, University of Dundee, Dundee, UK.

Abstract

CONTEXT:

The multiple mini-interview (MMI) is the primary admissions tool used to assess non-cognitive skills at Dundee Medical School. Although the MMI shows promise, more research is required to demonstrate its transferability and predictive validity, for instance, relative to other UK pre-admissions measures.

METHODS:

Applicants were selected for interview based on a combination of measures derived from the Universities and Colleges Admissions Service (UCAS) form (academic achievement, medical experience, non-academic achievement and references) and the UK Clinical Aptitude Test (UKCAT) in 2009 and 2010. Candidates were selected into medical school according to a weighted combination of the UKCAT, the UCAS form and MMI scores. Examination scores were matched for 140 and 128 first- and second-year students, respectively, who took the 2009 MMIs, and 150 first-year students who took the 2010 MMIs. Pearson's correlations were used to test the relationships between pre-admission variables, examination scores and demographic variables, namely gender and age. Statistically significant correlations were adjusted for range restrictions and were used to select variables for multiple linear regression analysis to predict examination scores.

RESULTS:

Statistically significant correlations ranged from 0.18 to 0.34 and 0.23 to 0.50 unrestricted. Multiple regression confirmed that MMIs remained the most consistent predictor of medical school assessments. No scores derived from the UCAS form correlated significantly with examination scores.

CONCLUSIONS:

This study reports positive findings from the largest undergraduate sample to date. The MMI was the most consistent predictor of success in early years at medical school across two separate cohorts. UKCAT and UCAS forms showed minimal or no predictive ability. Further research in this area appears worthwhile, with longitudinal studies, replication of results from other medical schools and more detailed analysis of knowledge, skills and attitudinal outcome markers.

© 2013 John Wiley & Sons Ltd.