변형된 면접: 학생선발을 위한 신뢰성 있는 면접의 부활? (Acad Med, 2012)

변형된 면접: 학생선발을 위한 신뢰성 있는 면접의 부활? (Acad Med, 2012)

Modified Personal Interviews: Resurrecting Reliable Personal Interviews for Admissions?

Mark D. Hanson, MD, MEd, Kulamakan Mahan Kulasegaram, Nicole N. Woods, PhD, Lindsey Fechtig, and Geoff Anderson, MD, PhD

특히 개별 면접은 전반적인 신뢰도가 낮은데 - 면접관 간 일치도가 낮고, 서로 다른 인터뷰 상황마다 일관성이 낮다 - 이로 인해서 예측력이 제한된다.

Particularly, personal interviews have low overall reliability— lack of agreement among interviewers and lack of consistency across different interview occasions—which, in turn, limits their predictive power.3,4

한 가지 흔한 해결책은 여러 차례 독립적인 샘플링을 하는 방법이다 (multiple independent sampling (MIS) method)

One common solution to increase the reliability of a performance measurement is to assess samples of the performance independently multiple times—that is, the multiple independent sampling (MIS) method.

가장 눈에 띄는 면접기법은 MMI이다. 고도로 구조화된, 시나리오 기반의 면접을 시행하는 방식이다. AAMC는 최근 MMI가 높은 신뢰도와 중간정도의 타당도를 가짐에도 불구하고, 그리고 일반적인 면접이 psychometric한 한계점이 있음에도 불구하고 개인면접을 시행하는 학교가 압도적임을 보고했다.

The most notable use of this measurement technique is the Multiple Mini-Interview (MMI),6,7 which uses up to 10 highly structured, scenario-based interviews to assess applicants. Interestingly, the Association of American Medical Colleges recently reported that a preponderance of schools use the admissions personal interview, not the MMI,1 despite not only the evidence regarding the MMI’s high reliability and moderate validity7–9 but also the critical psychometric limitations of the personal interview.2–4

MIS방법을 사용하는 비율이 낮은 이유는 상당한 자원이 투입되어야 하기 때문이며(신뢰도 있는 점수를 얻으려면 10명의 면접관이 필요하다) 면접관 모집에 관한 잠재적 영향 때문이다(모집 관련 활동과 관련된 변화). 추가적으로 기존 면접의 유연성과 직관적 단순함이 입학위원회가 MIS 도입을 꺼려하는 또 다른 이유이다.

Two critical factors contributing to the low uptake of the MIS method to admissions interviews are the aforementioned high resourcing requirements (10 interviewers needed to attain reliable scores) and the potential effects on recruitment (due to the associated alterations to campus recruitment-focused activities).10 Additionally, the flexibility and intuitive simplicity of the personal interview may make admissions committees (and interviewers) reluctant to abandon it all together.

Axelson과 Kreiter는 MIS의 적용을 연구했다. 2009년, 두 해 연속 면접을 본 지원자 집단을 대상으로 한 연구에서 전통적 면접 방식을 패널의 수를 줄이는 대신 독립적인 개인별 면접 수를 증가시킴으로써 신뢰도를 높일 수 있음을 보고했다. 따라서 입학위원회는 다수의 구조화된 시나리오에 의존하는 대신 다수의, 짧은, 단일 평가자로 이뤄진 면접을 수행함으로서 면접의 신뢰도를 향상시킬 수 있다.

Axelson and Kreiter10 investigated the application of MIS to the admissions personal interview itself. In their 2009 investigation, they reviewed the multiple interview scores of applicants who had been interviewed twice in consecutive years by a panel of two interviewers for admission to medical school. They estimated, after analyzing the scores of 168 candidates across four years who had interviewed twice, that reasonable reliability could be achieved using a traditional personal interview format by reducing the number of interviewers in the panel while increasing the number of separate personal interviews. Thus—instead of relying on a large number of structured scenarios—admissions committees might be able to depend on multiple, brief, single-rater interviews to enhance the reliability of the personal interview.

MPI에 대한 MIS 방법을 활용한 시도에 대한 연구

We report here the first prospective empirical test of the reliability of a similar modification to the admissions personal interview format using an MIS methodology named the modified personal interview (MPI).

방법

Method

1학년 학생에게 LEAD 프로그램에 대한 설명을 함. LEAD 지원자가 갖추어야 할 특성을 도출함. 이 특성에 대해 잠정적 지원자들과 communicate했으며, MPI과정동안의 질문을 만드는데 사용했다.

We informed the first-year students about LEAD and its selection process via announcements made during class and notifications sent over e-mail. The selection process constituted submission of written materials followed by, for a selected subset of candidates, the MPI process. We derived the attributes of successful LEAD candidates from the literature on leadership11,12 and through LEAD faculty consensus. These desired attributes were communicated to the pool of potential applicants and blueprinted onto (aligned with) questions asked during the MPI process.

제출 자료 Written submission materials (3 가지)

The written submission materials comprised three components: 

• a two-page curriculum vitae (CV) summarizing applicants’ academic and leadership experiences, 
• three brief descriptions of leadership experiences reported in the CV, and 
• a brief vision statement of leadership goals and career aspirations.

MPI 절차 The MPI process

4개 면접방, 10~12분, 4명의 평가자, 평가자들이 Behavioral description 질문 개발

Candidates who proceeded on to the interview stage moved among four interview rooms to complete the MPIs in succession. Each MPI was about 10 to 12 minutes long; a few interviews were longer at the discretion of the faculty interviewer. The four interviewers, all of whom had participated in the review of the written materials, framed all questions as behavioral descriptive questions which have strong validity in assessing personal characteristics.13–15

평가자들은 MPI 형식에 대해서 설명을 받고, 면접의 초점에 대해서도 연습함. 3개의 인터뷰는 반구조화되어있었으며, 평가자는 사전에 질문 목록을 가지고 있었음.

Interviewers received training on the MPI format and on the focus of the interviews. Three of the interviews were semistructured, and the interviewers used a list of predetermined questions.

4명의 평가자는 3개의 공통 특성과 한 개의 MPI-특이적 특성에 대해 평가함.

All four interviewers rated three common attributes—maturity, communication skills, and interpersonal skills—and a fourth attribute unique to their MPI.

평가자는 각 특성에 대해서 5점척도로 평가함. 총점 20점

The interviewers evaluated each attribute as a separate item on a five-point Likert- type scale (1 = poor, 2 = good, 3 = very good, 4 = excellent, and 5 = outstanding) to increase the scoring range available to interviewers. All items were totaled for a final MPI score out of 20, and overall total scores were used for selection.

Results

16명의 지원자, 10명에 대해서 MPI 수행, 8명 선발. 면접시간은 총 3시간

Sixteen candidates submitted initial applications to LEAD. Of these, we selected 10 for the MPI stage, 8 of whom were selected for the program. The entire set of MPIs was completed in three hours in one afternoon.

58%의 변인은 pi와 pq:i에 기인함.

The majority of variance among MPI scores (58%) was attributable to the participant–interview interaction (pi) as well as the participant–question interaction nested with MPI (pq:i), which suggests that these facets caused random error in the assessment of applicants.

전체 신뢰도는 0.79

Overall reliability of the MPI component and subsequent average MPI reliability was 0.79. The reliability of questions nested within MPIs (q:i) was 0.97.

Discussion and Conclusions

MIS가 MPI형태에 적용되었을 때 신뢰도가 높아진다. 4개의 MPI만으로도 0.7 이상의 신뢰도를 보여줌. 총 8 faculty hour 소모. 비슷한 수의 지원자를 대상으로 면접을 전통적 방식으로 한다면 13 faculty hours가 필요.

This report provides some evidence that MIS as applied within the MPI format is a reliable selection strategy. High reliability was achieved with just four MPIs, and a d-study revealed that future MPIs can achieve reliability greater than 0.7 with only three MPIs. A total of only 8 faculty hours was spent conducting the MPI process. A comparable traditional admissions personal interview of 40 minutes’ duration with two interviewers would take more than 13 faculty hours (66% more time) for the same number (n = 10) of applicants.

전통적 면접의 이러한 변형은 MIS 도입 가능성을 높여준다. MMI와 같이 기존의 MIS의 방식에 기반한 방식에서는 10명의 독립적 면접이 필요했다. 여기서 MPI는 3개의 인터뷰만으로도 threshold에 도달했다. 아마도 LEAD 선발 과정 때문일 수도 있다. 이러한 절차에서 사용된 MPI에는 좁은 범주의 특성만 평가했기 때문이다. 다른 비학업적 수행능력은 이미 의과대학 입학단계에서 평가되었다. 이러한 구체적인 제한적인 맥락이 면접 신뢰도를 높여줄 수 있다.

This modification of the personal interview has the potential to increase the uptake of MIS in admissions interviews. Previous application of MIS in the MMI showed that at least a minimum of 10 separate interviews were needed to achieve acceptable reliability.6,7 The MPI here met a minimum threshold at 3 interviews. A potential explanation for this finding is the specialized selection context of the LEAD admissions process. The MPI as used in this process focused on a narrow set of attributes related to leadership qualities as determined by LEAD faculty. Other aspects of nonacademic performance had already been assessed in the medical school admissions process. This specialized context also enabled the use of expert raters, which may have further enhanced interview reliability.

이러한 선발절차의 특수(전문)화는 안면타당도에도 기여한다. 안면타당도는 지원자가 지원절차가 직무에 관련되어있다고 믿는 정도라고 묘사되는데, (의과대학에서는 의과대학 교육과정 수행능력에 대한 추정가능성) 지원자가 면접 절차를 받아들이는 정도가 이 face validity와 관련되어있다.

The specialization of this selection process (with interviewers rating applicants’ performances according to a predetermined, defined suite of attributes aligned with a specific physician role—in this case, the role of physician leader) also lends to the face validity of the MPI format. Face validity has been described as the extent to which the applicants believe the application process is relevant to the job in question,16 or—to extrapolate to the medical school context—medical school curriculum. Applicant acceptance of admissions processes has been associated with face validity.16

본 연구에서 평가 특성의 오버랩은 content validity를 높여주었다. 지원서 점수와 MPI 점수의 상관관계가 높은 것은 LEAD 지원절차를 개발하는데 블루프린팅이나 매핑을 의도적으로 그렇게 한 것에 기인할 것이다. 지원서 평가자를 면접관으로 한 것 역시 영향을 주었을 수 있다.

In the current study, the overlap of attributes across the written application and MPIs enhanced content validity (and reliability). The strong association between written application scores and MPI performance is likely a result both of the intentional attribute mapping or blueprinting we performed in developing the LEAD application process and of the availability of written application materials during MPI occasions. The use of raters from the written application as interviewers may have also contributed to the strong association of scores across both evaluations, even though we removed all personal identifying information from the candidates’ written application materials.

Thus, we would not expect the recruitment of applicants to decrease through the use of the MPI format.

1 Dunleavy DM, Whittaker KM. The evolving medical school admissions interview. AAMC Analysis in Brief. 2011;11. https://www.aamc. org/download/261110/data/aibvol11_no7. pdf. Accessed June 13, 2012.

10 Axelson RD, Kreiter CD. Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ. 2009;43:1198–1202.

13 Taylor P, Small B. Asking applicants what they would do versus what they did do: A meta-analytic comparison of situational and past behaviour employment interview questions. J Occup Organ Psychol. 2002; 75:277–294.

15 Huffcut AI, Weekley JA, Wiesner WH, Degroot TG, Jones C. Comparison of situational and behavior description interview questions for higher-level positions. Pers Psychol. 2001; 54: 619–644.

---

Acad Med. 2012 Oct;87(10):1330-4.

Modified personal interviews: resurrecting reliable personal interviews for admissions?

Hanson MD1, Kulasegaram KM, Woods NN, Fechtig L, Anderson G.

Author information

• 1Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada. mark.hanson@utoronto.ca

Abstract

PURPOSE:

Traditional admissions personal interviews provide flexible faculty-student interactions but are plagued by low inter-interview reliability. Axelson and Kreiter (2009) retrospectively showed that multiple independent sampling (MIS) may improve reliability of personal interviews; thus, the authors incorporated MIS into the admissions process for medical students applying to the University of Toronto's Leadership Education and Development Program (LEAD). They examined the reliability and resource demands of this modified personal interview (MPI) format.

METHOD:

In 2010-2011, LEAD candidates submitted written applications, which were used to screen for participation in the MPI process. Selected candidates completed four brief (10-12 minutes) independent MPIs each with a different interviewer. The authors blueprinted MPI questions to (i.e., aligned them with) leadership attributes, and interviewers assessed candidates' eligibility on a five-point Likert-type scale. The authors analyzed inter-interview reliability using the generalizability theory.

RESULTS:

Sixteen candidates submitted applications; 10 proceeded to the MPI stage. Reliability of the written application components was 0.75. The MPI process had overall inter-interview reliability of 0.79. Correlation between the written application and MPI scores was 0.49. A decision study showed acceptable reliability of 0.74 with only three MPIs scored using one global rating. Furthermore, a traditional admissions interview format would take 66% more time than the MPI format.

CONCLUSIONS:

The MPI format, used during the LEAD admissions process, achieved high reliability with minimal faculty resources. The MPI format's reliability and effective resource use were possible through MIS and employment of expert interviewers. MPIs may be useful for otheradmissions tasks.

PMID:

 

22914517

 

[PubMed - indexed for MEDLINE]