MMI점수 타당화: 다양한 기질을 측정하는가? (Adv in Health Sci Educ, 2014)

MMI점수 타당화: 다양한 기질을 측정하는가? (Adv in Health Sci Educ, 2014)

Validating MMI scores: are we measuring multiple attributes?

Tom Oliver • Kent Hecker • Peter A. Hausdorf • Peter Conlon

도입

Introduction

MMI는 지원자의 비인지적 특성을 평가하기 위한 면접방법이다. 전통적인 면접(덜 구조화된)은 신뢰도와 타당도가 낮다고 보고되어 왔으며, MMI는 신뢰도가 충분히 높고, 의과대학 및 면허시험 수행능력과 유의한 상관관계가 있다. 추가적으로, MMI총저은 GPA와 같은 인지적 능력과 discriminant validity가 있어서, MMI가 뭔가 다른 것을 측정한다고 볼 수 있다.

The multiple mini-interview (MMI) is an interview method used in health professional school selection to assess the non-cognitive attributes of applicants (Eva et al. 2004). Whereas more traditional—and often less-structured—interviews have been found to have poor reliability and validity in health professional school selection (Kreiter et al. 2004; Eva et al. 2004; Albanese et al. 2003; Edwards et al. 1990), previous studies have found MMI scores to have sufficient reliability and to be significantly correlated to performance in school and licensure exams (Eva et al. 2009, 2012; Hecker and Violato 2011; Reiter et al. 2007). In addition, there is consistent evidence for the discriminant validity of total MMI scores from ratings of cognitive skill such as incoming grade point average (GPA; Eva et al. 2004, 2009, 2012; Reiter et al. 2007), which suggests that the MMI is measuring something other than cognitive skill.

MMI로 평가하는 비인지적 특성

Non-cognitive attributes assessed by the MMI

비인지적 특성에는 다양한 것들이 포함된다. MMI는 평가자가 한 스테이션 내에서 그리고 여러 스테이션을 거치며 지원자로부터 다양한 구인을 평가하게끔 설계되어 있다. 그러나 동일한 스테이션 내에서 평가한 서로 다른 종류의 비인지적 특성은 서로 상관관계가 매우 높은 것으로 보고되고 있어서, MMI점수는 흔히 각각의 구인에 기반한 점수가 아니라 총점을 활용하는 것이 일반적이다.

Non-cognitive attributes can include a variety of individual differences related to attitudes, personality traits, and motivations (Schmitt et al. 2009). MMIs have been designed to have raters assess candidates on multiple constructs (e.g. oral communication and moral rea- soning) both within and across interview stations. However, MMI measures of different non-cognitive attribute constructs assessed within the same station have been found to be highly correlated (Eva et al. 2004; Lemay et al. 2007; Roberts et al. 2009). As a result, it is common practice to report total scores (i.e. the average score across all measures) within each station instead of construct-based scores.

MMI가 측정할 수 있는 것에는 구두의사소통Oral Communication (OrCo)와 문제해결Problem Evaluation, PrEv가 있다. OrCo는 다양한 언어 메시지를 구조적으로 전달할 수 있는 능력이며, PrEv는 문제를 찾고 다양한 이해관계자들의 관점을 고려하여 의사결정과 판단을 내리는 능력이다.

Two of the more distinct interpersonal constructs that an MMI can attempt to measure are oral communication and problem evaluation. Oral communication is the ability to convey verbal messages constructively; and problem evaluation is the ability to identify and take into account multiple perspectives from various different stakeholders in decision making and judgment.

평가자들은 지원자의 OrCo와 PrEv 능력을 관찰하고 평가할 수 있다.

raters have an opportunity to observe and rate the candidate on

• the clarity of their language and confidence in their conveyed verbal response (oral communication), and

• the breadth and depth to which they can explore underlying issues within cases and correctly balance pros and cons for the situation (problem evaluation).

성격특성과 관련한 MMI 척도

MMI measures related to personality characteristics

세 가지 연구에서 MMI 총점과 성격척도의 관계를 살펴본 바 있다. 이들 연구의 결과는 혼재되어 있다.

Three exploratory studies have investigated the relationship between total MMI scores and personality measures (Griffin and Wilson 2012; Jerant et al. 2012; Kulasegaram et al. 2010). The results from these studies found mixed evidence

보건의료전문직의 대인관계 능력에 영향을 주는 두 가지 성격특성에는 emotionality와 extraversion이 있다.

Two personality traits that are likely to be related to health professionals’ interpersonal performance are emotionality and extraversion (Ashton and Lee 2007).

• Emotionality가 높은 사람: 공감을 잘 하고, 위험하거나 스트레스 상황에 민감하며, 다른 사람의 감정적 지지에 의지를 느낄 수 있다 People who have high emotionality tend to feel empathy and sentimental attachments with others, are sensitive to dangerous and stressful situations, and feel dependent on the emotional support from others;

• Extraversion이 높은 사람: 자신감이 넘치고, 그룹을 이끌며, 각종 모임과 관계를 즐기고, 열정과 에너지지가 넘친다 people who are extraverted tend to feel confident when leading or addressing groups of people, enjoy social gatherings and interactions, and frequently experience positive feelings of enthusiasm and energy.

외향성이란..

Extraversion is a trait that includes tendencies such as acting confident with others and expressing enthusiasm and energy (Ashton and Lee 2007).

정서성이란..

Emotionality is a trait that includes tendencies such as being familiar with the anxieties and fears that come with stressful situations, and feeling emotional connections with others (Ashton and Lee 2007).

미래 수행능력과 관계된 MMI 척도

MMI measures related to future performance

 

MMI에서는 다음을 측정(OrCo와 PrEv)

MMI measures of two distinct constructs (oral communication and problem evaluation) and

 

의사소통 스킬 인터뷰에서는 다음을 측정

communication skill interview scores of students’

• 효과적 관계구축 effectiveness in building a rela- tionship (i.e. build a patient’s or client’s feelings of rapport and trust with the practitioner) and

• 효과적 설명과 계획 effectiveness in explaining and planning (i.e. build a patient’s or client’s under- standing and motivation to support an action plan; Silverman et al. 2005).

OrCo는 효과적 관계구축과, PrEv는 효과적 설명 및 계획과 관계가 있을 것으로 생각함.

Oral communication should be more closely related to building a relationship and problem evaluation should be more closely related to explaining and planning.

연구목적과 가설

Research objectives and hypotheses

H1 Given the explicit measurement and distinctiveness of oral communication and problem evaluation, there will be a stronger model fit for a 2-factor solution for MMI scores than for a 1-factor solution. 

H2a Oral communication MMI scores will be positively related to building the rela- tionship score in a communication interview.

H2b Problem evaluation MMI scores will be positively related to explaining and plan- ning scores in a communication interview. 

H3a Oral communication MMI scores will be positively related to extraversion scores measured by the HEXACO-PI-R-60 (Ashton and Lee 2009). 

H3b Problem evaluation MMI scores will be positively related to emotionality scores measured by the HEXACO-PI-R-60 (Ashton and Lee 2009).

 

방법

Method

표본 Sample

척도 Measures

다면인적성면접

MMI

The MMI consisted of eight 10-min stations, with two raters per station who each inde- pendently rated the participants on two constructs. The development of the MMI followed the description outlined in Hecker et al. (2009). The majority of the stations were devel- oped at the University of Calgary, Canada and modified by the admissions committee at OVC.

The eight stations were meant to assess oral communication and problem evaluation for a range of issues relevant to success as a veterinarian. These issues were ethical and moral (2 stations), interpersonal (3 stations), intrapersonal (1 station), and professional (2 stations). At each station raters scored candidates on two items, one for each construct. Each item was scored on a scale of 1–5 (1 = unacceptable; 3 = meets expectations; 5 = exceptional)

커뮤니케이션 인터뷰

Communication interview

 

Adams and Ladner 가 설계한 표준화된 임상 커뮤니케이션 인터뷰가 있다. 각 참여자는 두 개의 인터뷰 스테이션에서 효과적인 의사소통 스킬의 활용을 평가받았다. 의학지식은 거의 필요하지 않았다. simulated client가 7개 항목에 대해서 즉각적으로 참여자를 평가하였다.(관계형성 4 문항, 설명과 계획 3문항)

The standardized clinical communication interviews were initially designed by Adams and Ladner (2004) with the consultation of practicing veterinarians. Each participant partici- pated in two communication interview stations designed to assess participants’ use of effective communication skills. Medical and technical knowledge requirements were minimal. The simulated client rated the participant immediately after each station on 7 items (using a 9 point scale) meant to assess two constructs, building the relationship (4 items) and explaining and planning (3 items).

 

두 점수는 T-score로 변환되어서 스테이션간 서로 다른 simlated client의 차이를 보정하고자 하였다. 두 스테이션의 T-score의 평균 점수를 계산하였다.

The two scores within each station were converted to a T-score to account for differences in simulated client scores between sta- tions (Howell 2002). The mean of each participant’s T-score across the two stations was calculated to use as his/her building the relationship and explaining and planning score.

성격

Personality

The personality traits of emotionality and extraversion were measured with the HEXACO- PI-R-60 (Ashton and Lee 2009).

분석

Analysis

 

결과

Results

고찰

Discussion

주요 결과

The major findings from this study were:

1. 2-요인 모델을 지지한다. 그러나 OrCo와 PrEv 구인은 두 모델에서 모두 상관관계가 매우 높았다.

1. There was support for a two factor model, however, the oral communication and problem evaluation constructs were highly correlated both within the model (.87) and the correlation analyses with the actual data (.73; Table 4).

2. OrCo 점수는 외향성, 그리고 관계구축과 유의한 상관관계가 있었다.

2. Oral Communication MMI score was significantly correlated with extraversion (small but significant) and building the relationship scores, supporting Hypotheses 2a and 3a.

3. PrEv점수는 정서성과 유의한 상관관계가 없었으나, 관계구축 및 설명과 계획과 유의한 상관관계가 있었다.

3. Problem evaluation MMI score was not significantly related to emotionality score but did correlate with building the relationship (not hypothesized) and explaining and planning, thus not supporting Hypothesis 2b but supported Hypothesis 3b.

4. MMI총점은 외향성과 작지만 유의한 상관관계가 있었고, 관계구축, 설명과 계획 과 유의한 상관관계가 있었다.

4. Total MMI score had a weak but significant correlation with extraversion, and significant correlations with building the relationship and explaining and planning.

2-요인 모델이 더 강력했으나, 두 요인의 상관관계가 높았다. 따라서 두 개의 truly distinct factor를 측정한다고 결론을 내리기에는 조심스럽다.

While there was a stronger and significantly better model fit for a two factor model (Fig. 1) than a one factor model, the two constructs were highly corre- lated (.87). Thus while there was support for a two factor model, caution must be taken in concluding that we are measuring two truly distinct factors as there was weak evidence for discriminant validity between the two construct scores.

본 연구결과의 실용적 의의를 찾자면,

practical implications of these findings : there is evidence for

• MMI 스테이션 구성에 시간과 노력을 투자할 가치가 있다. investing the time and effort in MMI station construction,

• 미래 수행능력을 예측하는 것으로 알려진 특성에 기반하여 평가표를 만들어야 함 creating appropriate scoring rubrics based upon attributes known to be pre- dictive for future performance and

• 평가자 훈련을 통해서 공정한 평가가 이뤄지도록 해야 함 conducting rater training to ensure appropriate and fair assessment of the candidate.

두 번째 연구의 목적은 MMI척도가 비인지적 구인의 nomological network에 부합하는지를 보는 것이었다. 흥미롭게도 OrCo와 PrEv의 MMI 점수가 매우 상관관계까 높았지만, 다른 비인지적 구인과의 상관관계는 서로 다르게 나타났다.

The second research objective was to test whether the MMI measures fit within the nomological network for non-cognitive constructs. Interestingly, even though the MMI scores of oral communication and problemevaluation were highly related, they were found to have different relationships to other measures of non-cognitive constructs.

2B가설과 같이, MMI의 PrEv점수는 '설명과 계획'과 유의한 관계가 있었다. 그러나 MMI의 PrEv점수에 대해서 정서성과 관련될 것이라는 가설은 맞지 않았다.

Consistent with our hypothesis (2B), the MMI problem evaluation rating had a sig- nificant positive relationship with explaining and planning. However, the hypothesized relationship between the MMI problemevaluation measure and emotionality was not found (hypothesis 3B). 

 

한 가지 설명은 MMI가 학생의 공감능력을 제대로 측정하지 못한 것이다. 현재의 MMI는 학생이 다른 사람의 관점을 얼마나 잘 인식하는지를 측정하지만, 얼마나 학생이 다른 사람에게 공감을 표현하는지는 측정하지 못한다. 좀 더 직접적으로 상호작용하는 스테이션을 포함시킴으로서 이러한 한계를 극복 가능할 것이다.

One explanation is that the MMI did not effectively measure students’ ability to empathize with others. The current MMI measured students’ ability to recognize the points of views of others, but it did not measure how students’ would express feelings of empathy towards others. One way that this could be done is to include stations that require candidates to engage more directly in an interaction.

또 다른 설명은 정서성이 광범위한 성격특성이라는 점이다. 광범위한 성격특성으로서 정서성은 성격의 여러가지 측면을 포함한다. 구체적인 준거와 강력한 개념상의 연결관계가 있을 때, Narrow trait가 Broad trait보다 종종 더 predictive한 것으로 알려져있다.

Another explanation is that emotionality is a broad personality trait. As a broad personality trait, emotionality measures a broad range of individual attributes (e.g. empathy towards others, sensitivity to physical harm). Narrow traits are often found to be more predictive than broad traits when there is a strong conceptual link to a specific criterion (Rothstein and Goffin 2006; Tett et al. 1991).

모든 valid한 선발과정은 그 과정을 거침으로서 일반적인 지원자 집단이 보다 균질한 집단으로 변해야 하나, 그러한 균질성은 학교나 직장에서의 성공과 관계된 특성에 대해서만 균질해야 한다.

Any valid selection process should lead to the selection of a more homogenous group of successful candidates from the general applicant pool, wherein the successful applicants are homogenous only on the characteristics that lead to in-school or in-job success.

따라서, 임상에서의 인터뷰 또는 health outcome을 더 향상시킬 수 있는 성격특성이 존재한다면, MMI시나리오가 그러한 것들을 평가할 수 있도록 설계되어야 한다.

Thus, if these are personality traits that can lead to better performance within the clinical interview and potentially better health outcomes, then it can be argued MMI scenarios should be designed to assess attributes related to these traits.

 

Ashton, M. C., & Lee, K. (2007). Empirical, theoretical, and practical advantages of the HEXACO model of personality structure. Personality and Social Psychology Review, 11, 150–166.

Jerant, A., Griffin, E., Rainwater, J., Henderson, M., Sousa, F., Bertakis, K. D., et al. (2012). Does applicant personality influence multiple mini-interview performance and medical school acceptance offers? Academic Medicine, 87, 1–10.

---

Adv Health Sci Educ Theory Pract. 2014 Aug;19(3):379-92. doi: 10.1007/s10459-013-9480-6. Epub 2014 Jan 22.

Validating MMI scores: are we measuring multiple attributes?

Oliver T1, Hecker K, Hausdorf PA, Conlon P.

Author information

• 1University of Guelph, Guelph, Canada.

Abstract

The multiple mini-interview (MMI) used in health professional schools' admission processes is reported to assess multiple non-cognitive constructs such as ethical reasoning, oral communication, or problem evaluation. Though validation studies have been performed with total MMI scores, there is a paucity of information regarding how well MMI scores differentiate the constructs being measured, the relationship between MMI scores (construct or total) and personality characteristics, and how well MMI scores (construct or total) predict future performance in practice. Results from these studies could assist with MMI station development, rater training, score interpretation, and resource allocation. The purpose of this study was to investigate the validity of MMI construct scores (oral communication and problem evaluation), and their relationship to personality measures (emotionality and extraversion) and specific scores from standardized clinical communications interviews (building the relationship and explaining and planning). Confirmatory factor analysis results support a two factor MMI model, however the correlation between these factors was .87. Oral communication MMI scores significantly correlated with extraversion (r c = .25, p < .05), but MMI scores were not related to emotionality. Scores for building a relationship were significantly related to MMI oral communication scores, (r c = .46, p < .001) and problem evaluation scores (r c = .43, p < .001); scores for explaining and planning were significantly related to MMI problem evaluation scores (r c = .36, p < .01). The results provide validity evidence for assessing multiple non-cognitive attributes during the MMI process and reinforce the importance of developing MMI stations and scoring rubrics for attributes identified as important for future success in school and practice.

PMID:

 

24449121

 

[PubMed - in process]