기관 내/기관 간 전통적 면접과 MMI의 신뢰도 비교: 다섯개 캘리포니아 의과대학(BMC Med Educ, 2017)

기관 내/기관 간 전통적 면접과 MMI의 신뢰도 비교: 다섯개 캘리포니아 의과대학(BMC Med Educ, 2017)

Reliability of Multiple Mini-Interviews and traditional interviews within and between institutions: a study of five California medical schools

Anthony Jerant1*, Mark C. Henderson2, Erin Griffin3, Julie A. Rainwater4, Theodore R. Hall5, Carolyn J. Kelly6,

Ellena M. Peterson7, David Wofsy8 and Peter Franks1

배경

Background

비구조화되었거나 최소한만 구조화된 일대일 전통적인 인터뷰(TI)는 의과대학 입학에 오랫동안 채용되어 왔다[1].

Unstructured or minimally structured one-on-one traditional interviews (TIs) have long been employed in medical school admissions [1].

부분적으로 면접관간 신뢰성에 대한 우려 때문에, 많은 학교들은 신청자들이 각각 다른 훈련된 평가관이 참여하는 일련의 간략한 반구조적 평가 스테이션을 통해 일하는 다중 미니 인터뷰(MMI)로 TI를 대체했다[3, 11].

Partly due to concerns about inter-interviewer reliability, many schools have replaced TIs with Multiple MiniInterviews (MMIs), in which applicants work through a series of brief, semi-structured assessment stations, each attended by a different trained rater [3, 11].

전술한 연구에 근거하여 일부 저자는 MMI가 TI에 비해 우수한 레이터 간 신뢰성을 가지고 있다고 결론지었다[2–6, 12, 17]. 그러나 이전의 MMI(및 TI) 연구는 각각 이러한 인터뷰 유형 중 하나만 채택하는 단일 기관에서 수행되었다.

Based on the foregoing studies, some authors have concluded that MMIs have superior inter-rater reliability as compared with TIs [2–6, 12, 17]. However, prior MMI (and TI) studies have been conducted at single institutions, each employing only one of these interview types.

중요한 것은, 두 가지 면접 유형을 모두 완료한 공통의 지원자 풀을 조사하여 TI보다 교수간 신뢰도가 높은지 여부를 동시에 테스트한 연구는 없다는 점이다. 더욱이 MMI 또는 TI의 학교 간 신뢰성을 조사한 연구는 없다.

Importantly, no studies have concurrently tested whether inter-rater reliability is higher for MMIs than for TIs by examining a common pool of applicants completing both interview types. Furthermore, no studies have examined the between-school reliabilities of MMIs or TIs.

5개의 캘리포니아 종방향 입학 평가(CA-LEAP) 컨소시엄 의과대학의 데이터를 사용하여 MMI와 TI의 내적 및 베트웨덴스쿨 신뢰성을 조사했다.

Using data from the five California Longitudinal Evaluation of Admission Practices (CA-LEAP) consortium medical schools, we examined the within and between school reliabilities of MMIs and TIs.

방법

Methods

연구대상

Study population

면접절차

Interview processes

두 학교(MMI1과 MMI2)는 MMI를 사용했으며, 각각 10분 스테이션과 7개 스테이션이 개별적으로 점수를 매겨 상업적으로 시판된 콘텐츠에서 채택되었다. [19] 두 학교 모두, 모든 스테이션은 다차원적이었다. 모든 스테이션에서 하나 이상의 추가 역량(예: 무결성/윤리, 전문성, 다양성/문화 인식, 팀워크, 스트레스 처리 능력, 문제 해결)과 함께 대인 커뮤니케이션 능력을 체계적인 평가 양식을 사용하여 평가하였다.

Two schools (MMI1 and MMI2) used MMIs, with 10 and 7 individually scored 10-min stations, respectively, generally adapted from commercially marketed content. [19] At both schools, all stations were multidimensional. Interpersonal communication ability was considered at every station, along with one or more additional competencies (e.g., integrity/ethics, professionalism, diversity/ cultural awareness, teamwork, ability to handle stress, problem solving), rated using a structured rating form.

3개 학교(TI1, TI2, TI3)는 TI를 사용했다. 각 학교에서 지원자들은 두 번의 30-60분짜리 구조화되지 않은 면접을 마쳤다. 하나는 교직원이고 하나는 의대생이나 교직원이었다.

Three schools (TI1, TI2, and TI3) used TIs. At each school, applicants completed two 30–60 min unstructured interviews, one with a faculty member and one with a medical student or faculty member.

TI1과 TI3 학교 모두에서 채택된 척도는 달랐지만 인터뷰 진행자는 단일 총괄 점수을 할당했다. 학교 TI2에서 면접관은 응시자를 4개의 개별 영역(생각/지식, 의사소통/행동, 에너지/인니티, 공감/동감)으로 1~5점 척도로 평가하였고, 그 후 도메인 점수를 합산하여 면접 점수(범위 4-20)를 산출하였다).

At both schools TI1 and TI3, interviewers assigned a single global interview rating, though the scales employed differed (

• exceptional, above average, average, below average, unacceptable at TI1; 

• unreserved enthusiasm, moderate enthusiasm, or substantial reservations at TI3). 

At school TI2, interviewers rated candidates on a 1– 5 point scale in four separate domains (thinking/knowledge, communication/behavior, energy/initiative, and empathy/compassion), and the domain scores were then summed to yield a total interview score (range 4–20).

Measures

Analyses

결과

Results

Within schools, correlations between interviewer ratings generally were qualitatively lower for TI1 (r 0.07, α 0.13), TI2 (r 0.29, α 0.40), and TI3 (r 0.44, α 0.61) than for MMI1 and MMI2 (α 0.68 and 0.60, respectively).

고찰

Discussion

우리는 일반적으로 TI보다 MMI가 학교 내 및 학교 간 신뢰성이 질적으로 더 높다는 것을 발견했다.

We generally found qualitatively higher within-school and between-school reliabilities for MMIs than for TIs.

본 연구에서 MMI의 질적으로 우수한 학교간 신뢰도에도 불구하고, 학교간 TI 신뢰성은 지배적인 견해를 바탕으로 우리가 예상했던 것보다 더 좋았다[2–6, 12, 17]. 이러한 연구 결과는 일부(전부는 아니지만) 이전의 단일 학교 연구에서 TI에 대해 관측된 낮은 인터뷰어 간 신뢰성이 TI 접근에 내재된 제한보다는 학교별 차이(예: 면접자 교육, 프로세스 표준화 정도)를 반영할 수 있음을 시사한다.

Despite the qualitatively superior between-school reliability of the MMI in our study, the between-school TI reliabilities were better than we had anticipated based on prevailing views [2–6, 12, 17]. These findings suggest that the low inter-interviewer reliability observed for TIs in some (but not all) prior single-school studies may reflect school-specific differences (e.g., interviewer training, degree of process standardization), rather than limitations inherent to the TI approach.

Conclusions

결론적으로, 5개의 캘리포니아 의과대학에 대한 공통 지원자 풀의 데이터를 분석한 결과, TI보다 MMI가 학교 내 및 학교 간 신뢰도가 정성적으로 높았다. 그럼에도 불구하고, TI의 학교 내 및 학교 간 신뢰성은 이전 문헌에 근거하여 일반적으로 예상보다 높았으며, 특히 특정 기관에서 사용하는 데 다른 요인이 유리할 경우 신뢰성에 대한 우려 때문에 TI를 포기할 필요가 없을 수 있음을 시사하였다.

In conclusion, in analyses of data from a common pool of applicants to five California medical schools, we found qualitatively higher within- and between-school reliabilities for MMIs than for TIs. Nonetheless, the within- and between-school reliabilities of TIs were generally higher than anticipated based on prior literature, suggesting that perhaps TIs need not be abandoned for the sake of reliability concerns, especially if other factors favor their use at a particular institution.

---

BMC Med Educ. 2017 Nov 6;17(1):190. doi: 10.1186/s12909-017-1030-0.

Reliability of Multiple Mini-Interviews and traditional interviews within and between institutions: a study of five California medical schools.

Jerant A1, Henderson MC2, Griffin E3, Rainwater JA4, Hall TR5, Kelly CJ6, Peterson EM7, Wofsy D8, Franks P9.

Author information

1

Department of Family and Community Medicine, University of California, Davis, School of Medicine, 4860 Y Street, Suite 2300, Sacramento, California, 95817, USA. afjerant@ucdavis.edu.

2

Office of the Vice Chancellor and Dean, University of California, Davis, School of Medicine, 4610 X Street, Suite 3101, Sacramento, California, 95817, USA.

3

Research and Evaluation Outcomes Unit, University of California, Davis, School of Medicine, 4610 X Street, Sacramento, California, 95817, USA.

4

Clinical and Translational Science Center, University of California, Davis, Health System, 2921 Stockton Boulevard, Suite 1400, Sacramento, California, 95817, USA.

5

Office of Admissions, David Geffen School of Medicine at University of California, Los Angeles, 885 Tiverton Drive Suite B27, California, Los Angeles, 90095, USA.

6

Division of Medical Education, University of California, San Diego, School of Medicine, 9500 Gilman Drive, mail code 0606, La Jolla, California, 92093, USA.

7

Office of Admissions, University of California, Irvine, School of Medicine, Medical Education Building, 836 Health Sciences Road, Irvine, California, 92697-4089, USA.

8

Office of Admissions, University of California, San Francisco, School of Medicine, Box 0408, 533, Parnassus Avenue, Room U-426, San Francisco, California, 94143, USA.

9

Department of Family and Community Medicine, University of California, Davis, School of Medicine, 4860 Y Street, Suite 2300, Sacramento, California, 95817, USA.

Abstract

BACKGROUND:

Many medical schools use admissions Multiple Mini-Interviews (MMIs) rather than traditional interviews (TIs), partly because MMIs are thought to be more reliable. Yet prior studies examined single-school samples of candidates completing either an MMI or TI (not both). Using data from five California public medical schools, the authors examined the within- and between-school reliabilities of TIs and MMIs.

METHODS:

The analyses included applicants interviewing at ≥1 of the five schools during 2011-2013. Three schools employed TIs (TI1, TI2, TI3) and two employed MMIs (MMI1, MMI2). Mixed linear models accounting for nesting of observations within applicants examined standardized TI and MMI scores (mean = 0, SD = 1), adjusting for applicant socio-demographics, academic metrics, year, number of interviews, and interview date.

RESULTS:

A total of 4993 individuals (completing 7516 interviews [TI = 4137, MMI = 3379]) interviewed at ≥1 school; 428 (14.5%) interviewed at both MMI schools and 687 (20.2%) at more than one TI school. Within schools, inter-interviewer consistency was generally qualitatively lower for TI1, TI2, and TI3 (Pearson's r 0.07, 0.13, and 0.29, and Cronbach's α, 0.40, 0.44, and 0.61, respectively) than for MMI1 and MMI 2 (Cronbach's α 0.68 and 0.60, respectively). Between schools, the adjusted intraclass correlation coefficient was 0.27 (95% CI 0.20-0.35) for TIs and 0.47 (95% CI 0.41-0.54) for MMIs.

CONCLUSIONS:

Within and between-school reliability was qualitatively higher for MMIs than for TIs. Nonetheless, TI reliabilities were higher than anticipated from prior literature, suggesting TIs may not need to be abandoned on reliability grounds if other factors favor their use.

KEYWORDS:

Interview as topic; Multiple mini-interview; Reproducibility of results; School admission criteria; Schools, medical

PMID:

 

29110662

 

PMCID:

 

PMC5674866

 

DOI:

 

10.1186/s12909-017-1030-0