인턴선발 MMI에서 과거행동면접 vs 상황면접 : 신뢰도와 수용가능도 비교(BMC Med Educ, 2015)

인턴선발 MMI에서 과거행동면접 vs 상황면접 : 신뢰도와 수용가능도 비교(BMC Med Educ, 2015)

Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliability and acceptability comparison

Hiroshi Yoshimura1,2,3*, Hidetaka Kitazono2, Shigeki Fujitani2, Junji Machi2,3, Takuya Saiki4, Yasuyuki Suzuki4 and Gominda Ponnamperuma5

세팅과 참가자

Settings and participants

TBIIMC 개요; 진료과; 미션; 교육목표; MMI 진행

TBUIMC is a Japanese general hospital, which newly introduced three specialty training programmes: internal medicine, surgery, and emergency medicine. To accomplish the trans-specialty mission of ‘fostering high-quality generalist physicians providing holistic patient care’, the educational committee of TBUIMC decided to introduce the Accreditation Council for Graduate Medical Education (ACGME) six general competencies [36] as educational outcomes. In 2013, the MMI took place at the partitioned TBUIMC conference room, in three separate weekends. Of the 26 candidates who applied for the TBUIMC programmes, 13, 10, and 3 were invited for the MMI on the first, the second, and the third day of the MMI, respectively.

면접 진행; 대상자; 면접관; 

Three separate days were set for candidates’ convenience, having better access to selection opportunities in TBUIMC; this facilitated the recruitment process. All candidates were Japanese medical graduates, whose level of training ranged from Post Graduate Year (PGY)-2 to PGY-4. They were either in the second year of, or had concluded the two-year National Obligatory Initial Postgraduate Clinical Training Programme (NOIPCTP), following their graduation from Japanese medical schools, and the Japanese National Licensure Examination [37]. A total of 18 examiners, including TBUIMC’s educational committee members (most of whom were US specialty board certified) and clinical supervisors, were all Japanese physicians in the aforementioned three specialties. All candidates, regardless of their applying specialties or the PGY level, were examined by all examiners, who were randomly allocated to the stations. All examiners stayed within the same station, on all three days.

인터벤션

Intervention

ACGME 여섯 개 역량 중 의학지식 제외; 나머지 다섯 개 역량은 하나당 한 스테이션; 각 역량당 2~8개의 하부 영역; 스테이션당 2명의 평가자; PBQ에서는 STAR Approach 사용. SQ에서 평가자는 독단적으로 probing은 못하게 함.

To base stations on the competencies of the ACGME, except ‘medical knowledge’, 5 stations were created to assess one competency (domain) per station. Out of the 2 to 8 sub-domains in each competency [36], two sub- domains (one for the PBQ, and the other for the SQ) per station were selected so that one PBQ followed by one SQ was administered within the same station (Table 1). The same questions were asked from all candidates. Two examiners were assigned to one station and they alternated questioning roles. 

• In PBQs, Situation-Task-Action-Result (STAR) approach was applied for guiding interviews [38]. 
• In SQs, presenting a scenario with a dilemma and making the candidates describe what they would do, in a situation where the candidate had to choose between two or more mutually exclusive courses of action [21,22] were followed by structured probing [27]. Examiners were not allowed to probe independently. 

A sample of instructions to exam- iners for one of the stations is shown in Table 2.

인터뷰가이드

10개 스테이션이면 충분히 reliable하다. 질문의 형태 외에도 다른 요인들이 영향을 미쳤을 것.

The current study suggests that less than 10 stations of the MMI with one examiner per station may be suffi- ciently reliable. In addition to the question format, other structuring processes may have contributed to this, e.g. 

• 기존에 확립된 프레임워크 basing stations on an established competency framework; 
• 불필요한 라포형성 최소화 minimising unnecessary rapport building between exam- iners and candidates; 
• 계획에 따른 동일한 질문 asking exactly the same questions from each candidate with planned probing; 
• 3개의 구분가능한 평가기준 활용 using three distinguishable rating rubrics; 
• 구체적 anchor에 따른 평가 rating candidates on points anchored with detailed descriptors; and 
• 평가자 훈련 providing exam- iner training. 

이러한 구조화 노력이 스테이션 수를 줄이는데 도움을 주었을 것임

These structuring efforts would help reduce the number of stations, especially where only limited examiner resources are available for a relatively smaller number of candidates.

평가자와 지원자가 긍정적(하지만 중등도의) 반응을 보인 것에는 스테이션 면접 형식이 이처럼 고도로 구조화된 것이 기여하는 바가 있을 것임. 흥미롭게도, 본 연구에서는 SQ와 PBQ에 대해 지원자와 면접관의 상반되는 반응을 보여준다. SQ는 지원자가 더 선호하였고, 평가자는 PBQ를 더 선호하였다. 특히 모든 참여자는 현재 MMI가 공평하며 SQ와 PBQ를 모두 사용하는 것의 중요성에 대해 언급하였다. 

As non-medical personnel selection studies have sug- gested [27], the highly structured nature of the station interview formats and other structuring efforts in the present study may be responsible for the positive but modest candidate and examiner reaction compared with previous studies [1,7-9,11-15]. Interestingly, this study also indicates contrasting acceptability for SQs and PBQs amongst candidates and examiners, i.e. SQs being more favourable for candidates as opposed to PBQs be- ing more favourable for examiners. Of particular note, all participants admitted fairness of the current MMI and most expressed importance of using both SQs and PBQs. As to how best PBQs and SQs could be com- bined, the participant reactions could be used as a guide for generating a discussion on both question formats at a given level (undergraduate or postgraduate [founda- tion, specialty, or subspecialty]) of admissions MMIs in the future, as is being discussed in the area of SSPIs in non-medical personnel selection [27].

---

BMC Med Educ. 2015 Apr 14;15:75. doi: 10.1186/s12909-015-0361-y.

Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliabilityand acceptability comparison.

Yoshimura H1,2,3, Kitazono H4, Fujitani S5, Machi J6,7, Saiki T8, Suzuki Y9, Ponnamperuma G10.

Author information

• 1Educational Committee, Prefectural Okinawa Nanbu and Children's Medical Centre, Haebaru Town, Okinawa Prefecture, Japan. yoshimura.hiroshi@gmail.com.
• 2Educational Committee, Tokyo Bay Urayasu-Ichikawa Medical Centre, Urayasu City, Chiba Prefecture, Japan. yoshimura.hiroshi@gmail.com.
• 3Department of Surgery, University of Hawaii, John A. Burns School of Medicine, Honolulu, State of Hawaii, USA. yoshimura.hiroshi@gmail.com.
• 4Educational Committee, Tokyo Bay Urayasu-Ichikawa Medical Centre, Urayasu City, Chiba Prefecture, Japan. hkitazono@gmail.com.
• 5Educational Committee, Tokyo Bay Urayasu-Ichikawa Medical Centre, Urayasu City, Chiba Prefecture, Japan. shigekifujitani@gmail.com.
• 6Educational Committee, Tokyo Bay Urayasu-Ichikawa Medical Centre, Urayasu City, Chiba Prefecture, Japan. junji@hawaii.edu.
• 7Department of Surgery, University of Hawaii, John A. Burns School of Medicine, Honolulu, State of Hawaii, USA. junji@hawaii.edu.
• 8Medical Education Development Centre, Faculty of Medicine, Gifu University, Gifu City, Gifu Prefecture, Japan. saikitak@gifu-u.ac.jp.
• 9Medical Education Development Centre, Faculty of Medicine, Gifu University, Gifu City, Gifu Prefecture, Japan. ysuz@gifu-u.ac.jp.
• 10Faculty of Medicine, University of Colombo, Colombo, Western Province, Sri Lanka. gomindap@hotmail.com.

Abstract

BACKGROUND:

The Multiple Mini-Interview (MMI) mostly uses 'Situational' Questions (SQs) as an interview format within a station, rather than 'Past-Behavioural' Questions (PBQs), which are most frequently adopted in traditional single-station personal interviews (SSPIs) for non-medical and medical selection. This study investigated reliability and acceptability of the postgraduate admissions MMI with PBQ and SQ interview formats within MMI stations.

METHODS:

Twenty-six Japanese medical graduates, first completed the two-year national obligatory initial postgraduate clinical training programme and then applied to three specialty training programmes - internal medicine, general surgery, and emergency medicine - in a Japanese teaching hospital, where they underwent the Accreditation Council for Graduate Medical Education (ACGME)-competency-based MMI. This MMI contained five stations, with two examiners per station. In each station, a PBQ, and then an SQ were asked consecutively. PBQ and SQ interview formats were not separated into two different stations, or the order of questioning of PBQs and SQs in individual stations was not changed due to lack of space and experienced examiners. Reliability was analysed for the scores of these two MMI question types. Candidates and examiners were surveyed on this experience.

RESULTS:

The PBQ and SQ formats had generalisability coefficients of 0.822 and 0.821, respectively. With one examiner per station, seven stations could produce a reliability of more than 0.80 in both PBQ and SQ formats. More than 60% of both candidates and examiners felt positive about the overall candidates' ability. All participants liked the fairness of this MMI when compared with the previously experienced SSPI. SQs were perceived more favourable by candidates; in contrast, PBQs were perceived more relevant by examiners.

CONCLUSIONS:

Both PBQs and SQs are equally reliable and acceptable as station interview formats in the postgraduate admissions MMI. However, the use of the two formats within the same station, and with a fixed order, is not the best to maximise its utility as an admission test. Future studies are required to evaluate how best the SQs and PBQs should be combined as station interview formats to enhance reliability, feasibility, acceptability and predictive validity of the MMI.

PMID:

 

25890189

 

[PubMed - indexed for MEDLINE] 

PMCID:

 

PMC4427914

 

Free full text