의과대학 학생선발에서 면접관의 편향 (The Medical Journal of Australia, 2010)

의과대학 학생선발에서 면접관의 편향

Interviewer bias in medical student selection

Barbara N Griffin and Ian G Wilson

OBJECTIVE:

면접관의 성격, 성별, 피면접자와 동성인지 여부, 면접관 훈련 등이 의과대학 학생선발 면접에 얼마나 영향을 주는가를 조사함.

To investigate whether interviewer personality, sex or being of the same sex as the interviewee, and training account for variance between interviewers' ratings in a medical student selection interview.

DESIGN, SETTING AND PARTICIPANTS:

2006년과 2007년 MMI에 참여한 지원자와 면접관을 대상으로 분석.

In 2006 and 2007, data were collected from cohorts of each year's interviewers (by survey) and interviewees (by interview) participating in a multiple mini-interview (MMI) process to select students for an undergraduate medical degree in Australia. MMI scores were analysed and, to account for the nested nature of the data, multilevel modelling was used.

MAIN OUTCOME MEASURES:

면접관이 준 점수, 지원자 점수의 분산

Interviewer ratings; variance in interviewee scores.

RESULTS:

2006년에는 153명의 면접관과 268명의 지원자가, 2007년에는 139명의 면접관과 238명의 지원자가 참여하였음. Agreeableness가 높은 면접관이 유의미하게 높은 점수를 주었으며, neuroticism이 높은 지원자가 낮은 점수를 주었다. 2006년에는 여성 면접관이 더 높은 점수를 주었다. 면접관과 지원자의 성별 조합에 따른 분산은 3.1%에서 24.8%까지 다양하였으나, 평균적 분산은 skills-based training이후에 크게 감소하였다.

In 2006, 153 interviewers (94% response rate) and 268 interviewees (78%) participated in the study. In 2007, 139 interviewers (86%) and 238 interviewees (74%) participated. Interviewers with high levels of agreeableness gave higher interview ratings (correlation coefficient [r] = 0.26 in 2006; r = 0.24 in 2007) and, in 2007, those with high levels of neuroticism gave lower ratings (r = -0.25). In 2006 but not 2007, female interviewers gave higher overall ratings to male and female interviewees (t = 2.99, P = 0.003 in 2006; t = 2.16, P = 0.03 in 2007) but interviewer and interviewee being of the same sex did not affect ratings in either year. The amount of variance in interviewee scores attributable to differences between interviewers ranged from 3.1% to 24.8%, with the mean variance reducing after skills-based training (20.2% to 7.0%; t = 4.42, P = 0.004).

CONCLUSION:

얼마나 점수를 잘 주는가가 면접관의 성격요인이나 성별에 따라 달라지지만 그 영향력은 작았다. 면접관을 무작위 배정하고, 비슷한 남-여 면접관을 배정하고, MMI를 활용하고, skill-based training을 하면 더 향상될 것이다.

This study indicates that rating leniency is associated with personality and sex of interviewers, but the effect is small. Random allocation of interviewers, similar proportions of male and female interviewers across applicant interview groups, use of the MMI format, and skills-based interviewer training are all likely to reduce the effect of variance between interviewers.

---

Interviewer personality

Interviewers completed the 20-item version7 of the International Personality Item Pool,8 measuring agreeableness, extraversion, neuroticism, conscientiousness and openness to experience. They were asked how accurately each item (eg, “sympathise with other’s feelings”) described them, using a scale from 1 for very inaccurate to 5 for very accurate.

Procedure

Applicants completed a 10-station MMI, which included one rest station. Each station lasted for 8 minutes and assessed a different quality. For example, Station 1 assessed applicants’ motivation to study medicine and Station 9 assessed communication skills. Interview format also varied; some stations involved sets of questions about past behaviour and experience (behavioural interviews), others presented scenarios or film clips for comment, and at Station 9 applicants were required to explain something to a “patient” (roleplayed by an actor). There was one interviewer per station. Ten applicants attended each MMI session and each interviewer worked for two sessions (ie, each interviewed 20 applicants).

All interviewers attended a 3-hour training session a month before the MMI. In 2006, the training was predominantly information-based, involving 2 hours of lecture about the rationale for including interviews in medical school student selection, information about the practical details of the MMI and how to score an applicant, the basics of behavioural interviewing, and instruction on avoiding bias. After a short break, the interviewers spent the remaining time in small groups practising using the rating scale and being given information about two MMI stations, with each small group studying different stations. 

Feedback from interviewers indicated that they wanted more skills training. Therefore, the 2007 training sessions were restructured to be predominantly skills-based training. Interviewers practised rating “simulated” interviewees, comparing outcomes and discussing examples of good and bad responses, and they interviewed trainers and each other to learn to probe appropriately. Notably, this training used the actual content of four of the nine stations (Stations 1, 3, 5 and 6). In addition, interviewers attended a half-hour briefing immediately before interviewing at the 2007 MMI sessions, when they were given individual training on the content of the specific station they would be attending.

Analysis

It is essential to use multilevel modelling to account for the nested nature of the interview datasets on which studies such as ours are based.9 When interviewees are rated by a subset of interviewers, they are “nested” under that subset. Analyses that disregard this multilevel component ignore dependencies between variables, artificially reduce standard errors and introduce correlated prediction errors. Not only does this violate statistical assumptions (eg, independence), but it increases the chance of finding significant results related to interviewer variables and decreases the chance of finding significant results related to individual (applicant) differences. Hierarchical linear modelling was therefore used (HLM 6.6 [SSI Scientific Software International, Lincolnwood, Ill, USA]), in addition to correlations and t tests for comparison of means. The threshold of significance was set at P = 0.05. The research was approved by the institution’s Human Research Ethics Committee.

---

Med J Aust. 2010 Sep 20;193(6):343-6.

Interviewer bias in medical student selection.

Griffin BN1, Wilson IG.

Author information

• 1Psychology, Macquarie University, Sydney, NSW, Australia. barbara.griffin@mq.edu.au

Erratum in

• Med J Aust. 2010 Oct 18;193(8):486.

Abstract

OBJECTIVE:

To investigate whether interviewer personality, sex or being of the same sex as the interviewee, and training account for variance between interviewers' ratings in a medical student selection interview.

DESIGN, SETTING AND PARTICIPANTS:

In 2006 and 2007, data were collected from cohorts of each year's interviewers (by survey) and interviewees (by interview) participating in a multiple mini-interview (MMI) process to select students for an undergraduate medical degree in Australia. MMI scores were analysed and, to account for the nested nature of the data, multilevel modelling was used.

MAIN OUTCOME MEASURES:

Interviewer ratings; variance in interviewee scores.

RESULTS:

In 2006, 153 interviewers (94% response rate) and 268 interviewees (78%) participated in the study. In 2007, 139 interviewers (86%) and 238 interviewees (74%) participated. Interviewers with high levels of agreeableness gave higher interview ratings (correlation coefficient [r] = 0.26 in 2006; r = 0.24 in 2007) and, in 2007, those with high levels of neuroticism gave lower ratings (r = -0.25). In 2006 but not 2007, female interviewers gave higher overall ratings to male and female interviewees (t = 2.99, P = 0.003 in 2006; t = 2.16, P = 0.03 in 2007) but interviewer and interviewee being of the same sex did not affect ratings in either year. The amount of variance in interviewee scores attributable to differences between interviewers ranged from 3.1% to 24.8%, with the mean variance reducing after skills-based training (20.2% to 7.0%; t = 4.42, P = 0.004).

CONCLUSION:

This study indicates that rating leniency is associated with personality and sex of interviewers, but the effect is small. Random allocation of interviewers, similar proportions of male and female interviewers across applicant interview groups, use of the MMI format, and skills-based interviewer training are all likely to reduce the effect of variance between interviewers.

PMID:

 

20854239

 

[PubMed - indexed for MEDLINE]