의과대학 입학에 있어서 인적특성 평가(Med Educ, 2005)

Assessment of personal qualities in relation to admission to medical school

Mary Ann Lumsden,1 Miles Bore,2,3 Keith Millar,4 Rachael Jack1 & David Powis2

의과대학 학생선발 절차의 목적은 학부의학교육을 잘 이수하여 미래에 좋은 의사가 될 사람을 뽑고, 전문직에게 악평을 남길 사람을 배제하는 것이다.

The aim of a medical school admissions procedure is to select those who will perform well as undergraduates and make good doctors in the future, and to exclude those who will bring the profession into disrepute.

추가로, 의과대학 졸업생의 사회적, 문화적, 인종적 배경 구성이 어떻게 되어있느냐는 전체 환자집단의 다양성을 반영해야 한다.

In addition, the social, cultural and ethnic backgrounds of medical graduates should reflect the diversity of the patient population.

고등학교졸업생을 대상으로 한 영국의 의과대학생 선발은 높은 학업성취도를 이룬 학생을 주로 선발하며, 그 절차는 비밀스럽고 다양하다.

The admission of school-leavers to medical schools in the UK is usually based upon high academic attain- ment, the procedures often being secretive and varied.2

그러나 학업성취도가 높은 학생을 선발하는 것은 그 성장배경이 애초에 불리한 사람들을 차별하는 결과를 가져올 수 있다. 그리고 이는 지원자와 교사 모두의 의욕을 꺾는 일이다. 더 나아가서 불리한 성장배경에서 자라난 지원자는 현재의 의과대학 선발요건을 달성하기 어려운 경우가 많은데, 이는 그들의 학업적 역량을 최대로 발휘할 기회가 주어지지 않았기 때문이다.

However, high academic thresholds may discriminate against those from disadvantaged backgrounds7,8 as these discourage both applicants and teachers. Furthermore, applicants from disadvantaged backgrounds find current admissions requirements difficult to attain because they often lack opportunity or encouragement to maximise their academic potential. 

더 광범위하게는, 의사들은 우수한 역량과 더불어 공감과 윤리성을 갖추어야 한다. 

More broadly, it is generally accepted that doctors need to be competent, empathic and ethical. (Leaving aside particular psychomotor skills, generic competence in this context refers to the intellectual ability to solve problems by applying acquired knowledge and using logical reasoning.)

psychometric 검사는 학습자료 이외에서 드러나는 인적특성을 측정하는 도구로 사용될 수 있고, 따라서 이는 사회적 배경에 영향을 덜 받는다.

Psychometric tests can be used to measure personal- ity characteristics and abilities rather than learned material, and accordingly, performance should be less influenced by social background.

의과대학 지원자에 대한 Psychometric testing은 미국과 호주에서 몇 년간 사용되어왔다. 인지능력은 그 자체로 혹은 지원자의 다른 학업기록과 함께 사용될 경우에도 의과대학 수행능력의 신뢰도 높은 예측인자임이 확인되어왔다.

Psychometric testing of applicants to medicine has been used in the USA and Australia for some years. Cognitive ability scores alone, and in combination with the academic record of the applicant, have been shown to be reliable predictors of medical school performance.10

PQA가 호주에서 개발되었으며 세 가지 검사를 포함하고 있다.

A test battery called the Personal Qualities Assess- ment (PQA) has been developed by researchers in Australia. The battery consists of 3 tests, of which the 

• first is designed to measure individual differences in cognitive reasoning ability, 
• the second to identify an Involved (empathic and confident with others) or Detached (narcissistic and aloof) personality trait, and 
• the third to determine ethical ⁄ moral orientation.

PQA에 대한 추가 정보

Further details are available from the PQA website.13

인성검사과 윤리/도덕 지향검사는 그러한 특성이 강하게 발전된 개개인을 대상으로 활용될 수 있다. 예컨대 그 검사를 통해서 공감이 높은 사람 혹은 자기확신이 강한 사람을 찾을 수 있으며, 자기애적인 사람과 무관심한 사람을 찾을 수 있다. 이 두 가지 검사도구 결과를 병합하면 Table1 과 같은 결과가 나온다.

The personality and ethical ⁄ moral orientation tests can be used to identify individuals in whom certain characteristics are strongly developed. For example, they allow for identification of those who are very empathic or self-confident, as well as those who are particularly narcissistic or aloof. In addition, the scores from these 2 assessment tools can be combined, enabling an empirically-based definition of different personality types (Table 1).

요약하자면 PQA는 다음과 같은 것을 검사한다.

In summary, the PQA is used to determine: 

• cognitive reasoning ability; 

• the degree to which individuals are involved or detached, and 

• the extent to which individuals value individual freedom at the expense of societal needs or the needs and expectations of society at the expense of the individual.

The tests ⁄ questionnaires

Participants completed the following tests.

• Test 1: Cognitive Skills (TUNRA, University of Newcastle, Australia): This comprised a 38-itemtest to measure the ability to reason logically and to apply logic to problemsolving.

• Test 2: Mojac Scale11,14–16: This provides a relative measure of a libertarian-dual-communitarian dimension of moral ori- entation (LibCom score).

• Test 3: NACE⁄ (ECAN) Scale12,16: This provides scores on narcissism(N), aloofness (A), self-confidence(C) andempathy (E). Anoverall ECAN score was calculated: highscores indicate a tendency to be involved with others (high empathy and self- confidence) while lowscores indicate a tendency to be detachedfromothers (highaloofness andnarcissism).

도덕성 유형 

Moral types

이 연구를 위해서 평균에서 1.5SD 떨어진 것을 임의로 기준으로 삼았다.

For this research the types were defined arbitrarily as a score on either dimension greater than 1.5 standard deviations from the mean. The raw scores were converted to T scores, having a mean of 50 with an SD of 10. Accordingly, outlying scores lie below 35 or above 65 on either dimension.

통계 분석

Statistical analysis

Results are expressed as means (SD) for normally distributed data. The Kruskall)Wallis and Mann–Whitney tests were used to examine non-parametric data.

부모의 직업 자료는 얻는 것이 불가능해서 DepCat 분류를 사용했다.

As data on parental occupation were not available, social class was defined by deprivation category (DepCat), whereby the individual’s postcode serves as a proxy for socioeconomic class.19

1~7까지이며, 1,2 / 3,4,5 / 6,7로 구분하였음.

DepCats range from 1 to 7, the latter indicating areas of severe deprivation. DepCat data were available on 427 individuals. Due to the small numbers inDepCat 6and7particularly, the cohort was divided into 3 groups for analysis.19 Group 1 comprisedthoseinDepCats 1and2 (n ¼ 164, 38.4%); group 2 comprised those in DepCats 3, 4 and 5 (n ¼ 229, 53.6%) and group 3 comprised those in DepCats 6 and 7 (n ¼ 34, 8.0%).

PQA 프로파일

Profile of the PQA

인지기술

Cognitive skills

The range and normal distribution of the scores shown in Fig. 1 indicate that there are widely differ- ent cognitive abilities in this sample, which is other- wise uniformin terms of its members having achieved high grades in school-leaving examinations.

MOJAC과 NACE 점수

MOJAC and NACE scores

Figure 2 shows the distribution of applicants by moral type:

Gender

Educational background

Deprivation category (DepCat)

선발절차에 PQA를 포함시키는 것의 효과

The effect of incorporating the PQA into theselection process 

평균보다 2SD이상 떨어지는 지원자는 의학에 부적합하다고 볼 수 있다. 이 지원자들을 배제하고, 성격검사에서 극단에 있는 사람들을 제외한다면, 109명의 학생이 남는다. 그러나 이 중에 실제로는 15명만이 선발단계에서 배제되엇다.

It may be hypothesised that those applicants withrelatively poor cognitive skills (> 2 SD below thecohort mean score) are less suited to a career inmedicine. If these individuals are excluded, together with those whose personality traits lie at the extreme ends of the scale, 109 (23%) of the Scottish applicants would not have obtained a place. Only 15 of these applicants were in fact excluded on the standard admissions criteria applied. Figure 4 shows the actual acceptance and rejection rates of the present sample in their applications to study medicine, and their application outcomes if the PQA or standard criteria had been applied.

DISCUSSION

인지기술 점수는 가우시안 분포를 따랐으며, 이 전체 지원자의 학업성취가 전반적으로 극도로 높음에도 이러한 인지적 능력에는 상당한 차이가 있는 것을 보여준다. 이는 'A레벨을 받는 것'자체로는 - 이미 그러한 학생들이 수가 늘어나서 - 충분히 쓸모있는 선발기준이 되지 못한다는 여러 기관의 우려와 일치한다.

The scores of the cognitive skills test component of the PQA follow a Gaussian distribution and it is possible to distinguish different degrees of cogni- tive ability despite the fact that academic attain- ment across the cohort is extremely high. This is an important finding in light of the concerns expressed by both the media and organisations such as the Council of the Heads of Medical Schools that A-levels are becoming less useful as a selection criterion because increasing numbers of candidates achieve the top grades.

인지적 능력 뿐 아니라 PQA는 NACE과 Mojac도 확인한다. PQA에서 드러난 특성은 교육적 배경에 따라 영향을 받지 않으나, 일부 DepCat에 따라서 약간의 차이는 있다. 더 Deprivation이 심한 집단에서 ECAN점수가 높으며, 이는 이들이 더 공감적이고 자신감이 있으며, 덜 자기애적이고 덜 무관심하다는 것을 보여준다.

As well as cognitive ability, the PQA has also been developed to identify empathy and related and opposing traits (NACE) and moral ⁄ ethical orienta- tion (Mojac). The qualities identified by the PQA were not influenced by educational background, although some minor differences with deprivation category were noted: those from areas of greater deprivation produced higher ECAN scores, indica- ting that they were more involved (empathic and confident) and less detached (narcissistic and aloof) than those from areas of lesser deprivation.

13 Personal Qualities Assessment (PQA). http://www. pqa.net.au

Med Educ. 2005 Mar;39(3):258-65.

Assessment of personal qualities in relation to admission to medical school.

의과대학 입학생의 성공에 필요한 핵심 인성역량 : 무엇이고, 어떻게 입학단계에서 평가할 수 있는가?(Acad Med, 2013)

Core Personal Competencies Important to Entering Students’ Success in Medical School: What Are They and How Could They Be Assessed Early in the Admission Process?

Thomas W. Koenig, MD, Samuel K. Parrish, MD, Carol A. Terregino, MD, Joy P. Williams, Dana M. Dunleavy, PhD, and Joseph M. Volsch, MPA

입학시에 의과대학 입학생이 지녀야 할 학업적 역량에 대해서는 일반적 합의가 있다.

There is general agreement in the medical education community about the academic competencies that medical students should demonstrate when they matriculate.

비록 의학교육계는 졸업시에 갖춰야 하는 인성역량에 대래서는 동의를 이뤘으나, 입학시에 어떠해야 하는가에 대해서는 합의가 없다.

Although the community has agreed on the personal competencies that medical students should demonstrate at graduation,1 it has not reached consensus on those that are important at entry or how to incorporate them into the admission process.

Albanese and colleagues2 estimated that more than 87 different personal qualities are assessed during the admission process. This lack of consensus among schools is surprising given that research has linked certain personal competencies to positive admission and medical school outcomes.

연구 결과를 살펴보면 어떤 인성 특징든 환자 성과, 환자들의 의사에 대한 평가에도 연관된다고 했다.

Researchers have related some of these personal characteristics and skills to improved patient care outcomes and to patients’ ratings of their physicians.5,6

관련된 연구 결과로는 다음과 같은 것이 있다.

• For example, good teamwork and collaboration are correlated with improved patient outcomes, patient satisfaction, and greater job satisfaction among physicians.7 
• Patients who report being treated with dignity by their physicians are more likely to adhere to treatment plans and to be satisfied with their care.8 
• Similarly, physicians who “adopt a warm, friendly, and reassuring manner” with their patients are more effective than those who keep consultations formal and do not offer reassurances.9 
• Recently, Hojat and colleagues10 found that patients of physicians with high levels of empathy have better health outcomes than patients of physicians with moderate and low levels of empathy. 
• Moreover, when physicians’ personal skills are lacking, negative professional outcomes are likely. 
• For instance, Papadakis and colleagues11 showed that unprofessional behavior in medical school (e.g., irresponsibility, lack of capacity for self-improvement) predicts later disciplinary action by state medical boards.

의과대학 입학에서 인성역량의 역할

The Role of Personal Competencies in Medical School Admissions

AAMC의 리더들과 다른 의학교육계 사람들은 지원자의 인성 특성을 더 강조해야 한다고 말했다.

Leaders of the Association of American Medical Colleges (AAMC)12 and others in the medical education community have called for more emphasis to be placed on applicants’ personal competencies in the admission process.

자료를 보면 면접까지 가기 전에 상당한 수의 지원자가 제외된다. 2011년, 한 지원자는 평균 14개의 의과대학에 지원했으나 평균적으로 2개 이하의 학교에서만 면접까지 갔다.

Data show that a significant part of admission screening takes place before interviews: In 2011, the average applicant submitted 14 applications but received less than 2 interview invitations.13

이 문제를 해결하기 위해서 가장 먼저 해야 할 것은 평가할 인성역량을 정하고, 의학교육계의 수요와 목적의 균형에 맞는 실무적, psychometric적 이슈를 해결하기 위한 도구에 합의해야 한다.

To meet this challenge, the medical education community must first agree on a universal set of personal competencies to measure as well as a set of tools that balances the needs and goals of the admission community with practical (e.g., cost, accessibility) and psychometric issues.

Defining Core Personal Competencies for Entering Medical Students

Although there have been attempts to systematically define the personal competencies that medical school matriculants should demonstrate on entry (e.g., the AAMC explored this in the 1970s and 1990s),16 there has been little effort to build consensus about these competencies in the wider medicaleducation community. Therefore, the AAMC undertook a rigorous, multi- year process to research and identify core personal competencies for students entering medical school in the 21st century. This process (Table 1) 

16 Etienne PMJ, Julian ER. Assessing the personal characteristics of premedical students. In: Camara WKE, ed. Choosing Students: Higher Education Admissions Tools for the 21st Century. Mahwah, NJ: Lawrence Erlbaum Associates; 2005.

의과대학 성공에 중요한 인적 특성

Identifying personal characteristics important to success in medical school

MR5 Committee는 두 개의 설문 진행

The MR5 Committee began the process by conducting two surveys

• In 2008, U.S. and Canadian admission officers were asked to describe their school’s admission process and to rate the importance of 41 personal characteristics to success in medical school.
• In 2009, U.S. and Canadian academic affairs officers were asked to rate the importance of 72 characteristics to success in medical school. Data from these two surveys17,18

핵심 인성역량 개발

Developing the set of core personal competencies

평균적으로 매우-극도로 중요하다고 평가하였으나, 입학과정에서 관련되어 제공되는 정보의 질에는 만족하지 못함

As shown in Table 2, on average, all of the draft personal competencies were rated by admission officials as “very important” to “extremely important.” Respondents were not, however, satisfied with the quality of information available about these competencies during the admission process.

피드백 수집

Collecting feedback on the core personal competencies

The ILWG’s recommendation served as the foundation for the AAMC Admissions Initiative. One of that group’s first projects was to review the ILWG’s draft definitions of the recommended core personal competencies.

9개의 핵심인성역량 승인

Approving the nine core personal competencies for entering medical students

2013년 2월 AAMC COA는 최종 9개 리스트를 만들었다. 

In February 2013, the AAMC’s COA endorsed the final list of nine core personal competencies for entering medical students (defined in Table 4): 

• ethical responsibility to self and others; 
• reliability and dependability; 
• service orientation; 
• social skills; 
• capacity for improvement; 
• resilience and adaptability; 
• cultural competence; 
• oral communication; and 
• teamwork.

평가도구 검색

Exploring Tools to Assess the Core Personal Competencies Early in the Admission Process

비록 ILWG 설문이 인성역량을 평가하는 도구도 제안하긴 했지만, 그것의 활용과 가치에 대해서는 아직 답해야 할 질문이 많다.

Although the ILWG survey suggested a desire among admission officers for tools that assess applicants’ personal competencies early in the admission process, there are many unanswered questions about the use and value of such measures in medical school admissions.22

50개 이상의 논문, 6개의 미발표된 보고서 등등을 보았음.

We identified more than 50 seminal articles (including several meta-analyses) and six nonpublished technical reports about tools currently used to measure personal competencies in higher education and employment settings. We made subjective, holistic judgments about tools’ potential to provide information on applicants’ core personal competencies for use in the pre-interview screening stage of the admission process. 

여섯 개의 도구를 여덟 개의 준거로 평가하였다.

We judged six types of tools according to the following eight criteria: 

• validity, 
• reliability, 
• group differences, 
• susceptibility to faking and coaching, 
• applicant reactions, 
• user reactions, 
• cost/resource utilization, and 
• scalability for use in pre-interview screening (Appendix 1).

SJT수행에 약간의 인종/민족간 차이가 있을 수 있다. 그러나 법과대학 결과를 보면 AA나 Latino의 입학을 더 높여준다.

There is some evidence suggesting that there may be small racial/ethnic group differences in performance on SJTs that emphasize decision making.31 However, research conducted by the College Board and the Law School Admission Council indicates that including these tests in the admission process may increase the percentage of African American and Latino matriculants compared with using academic data alone, and that performance on SJTs is the best predictor of “lawyering effectiveness.”32,33

SJT는 개발에 비용이 많이 들며, 시나리오 개발에 기술전문가가 필요하다. 그러나 면접 전 스크리닝에 scalable하며 대규모의 지원자에게 시행할 수 있다. 결과 자료도 활용하기 용이하다.

SJTs are somewhat expensive to develop because of the technical expertise needed to create and score scenarios. However, they are scalable for use in pre-interview screening because they can be administered to a large number of applicants before the interview. Further, when SJTs are scored, data are presented in a format that is easy to consume.

표준화 수행 평가(SEP)

Standardized evaluations of performance

평가자는 그래픽, 상호비교, 행동-관련-평가스케일을 활용하여 지원자를 특정 역량에 따라 평가한다.

In standardized evaluations of performance (SEPs), raters use a graphic, comparative, or behaviorally anchored rating scale to evaluate applicants on a set of competencies.

대부분의 의과대학 입학절차에서 비표준화 추천서를 사용하지만, 비학업적 변인에 대해서는 평가자간 신뢰도가 낮고, 예측 타당도도 낮고, comparative data도 부족하다. 다른 전문직 교육과정에서는 SEP을 사용해왔으며 ETS에서는 PPI를 도입했다. 

Although most medical school admission processes use nonstandardized letters of recommendation—which have poor interrater reliability for nonacademic variables,34 have poor predictive validity, and lack comparative data—other graduate and professional program (e.g., veterinary medicine, optometry, physical therapy) admission processes use SEPs. In 2009, the Educational Testing Service introduced the Personal Potential Index,35 an SEP for use in graduate admissions, but there is no published literature to date on its psychometric properties.

의과대학 지원자의 수행능력을 보면 SEP 점수와 작지만 유의미한 정의 상관관계를 확인할 수 있다. 입학담당자는 SEP에 긍정적인 태도를 가질 가능성이 높은데, 왜냐하면 평가자들이 지원자의 인성역량을 묘사할 구체적인 행동 예시를 포함시켜야 하기 때문이다.

Research on the Medical Student Performance Evaluation shows small but significant observed positive correlations between standardized evaluations and performance on comprehensive clinical performance exams.36 Admission officers are likely to have positive attitudes about SEPs because raters must include specific examples of behaviors illustrating applicants’ personal competencies.37

점수를 잘 주는 평가자가 있을 수 있고 평가에 variance가 있을 수 있다.

There is potential for rater leniency and consequent lack of variance in ratings, though.

신뢰도는 이러한 성취기록이 감독관이 있는 상황에서 수집되고, 다수의 평가자가 평가하여 확보할 수 있다. 타당도 자료는 수집 불가능하다. 지원자와 의과대학은 미적지근할 수 있는데 일이 늘어나기 때문이다. 점수를 부여하지 않은 성취기록에 대해서는 거의 문헌자료가 없으나 개발이 어렵지 않고 다수의 지원자에게 적용가능하다.

Reliability is best when accomplishment records are collected in proctored settings and are scored by multiple raters.39 Validity data are not available with respect to their use in admissions. Applicants and users may have lukewarm reactions to them because of the added workload. There is little published research on unscored accomplishment records, but they are inexpensive to develop and can be administered to large numbers of applicants.

면접

Local interviews

인터뷰 종류는 비구조화에서 구조화까지 다양하나 대부분 반구조화 면접을 사용한다.

Interview types range from unstructured to structured, but most medical school interviews are semistructured.

면접은 한계가 많은데, 우선 비구조화 면접의 신뢰도가 낮으며, 지원자의 지원서 자료를 같이 면접관에게 제공하는 평가에 오류가 생길 가능성이 높다. 또한 평가자 오류가 있을 수 있고, 지원자보다 평가자에 따라 점수가 달라진다.

Local interviews have a number of limitations, however. Reliability for unstructured interviews is poor, and the practice of providing interviewers with access to applicants’ application data introduces bias.42,43 In addition, local interviews are subject to rater error, and ratings may have more to do with the interviewer than the interviewee.43 

구조화된 면접은 다음과 같은 것이 있다.

structured interviews conducted at the University of Iowa Carver College of Medicine,47 and “behavioral event interviews” used by the Scholarly Excellence, Leadership Experiences, Collaborative Training program at the Morsani College of Medicine48

우리의 권고는 SJT, SEP, 성취기록이다.

After reviewing the literature and evaluating potential tools on the eight criteria, we suggested that the AAMC further investigate three tools for possible use in assessing applicants’ core personal competencies during the admission process: SJTs, SEPs, and accomplishment records. We recommended these tools because each of them

그 이유는 아래와 같다.

• provides data about personal competencies in a format that is easy to use and would be available in time for pre-interview screening, 

• allows for multiple sources of assessment, 

• has acceptable validity and is likely to provide predictive value beyond UGPAs and MCAT scores in predicting nonacademic outcomes, 

• demonstrates less potential risk of coaching and faking effects compared with other tools, 

• is likely to be accepted by applicants and admission officers, and 

• avoids exorbitant costs that would likely be passed on to applicants.

어떤 도구도 모든 상황에 완벽하지 않아서 다양한 도구를 사용할 것을 권고한다. SJT SEP 성취기록은 모두 toolbox로서 함께 사용되어야 한다.

No tool is perfect for all situations, so we recommend that multiple tools be employed to assess personal competencies to enable admission officers to evaluate the information collected (just as they currently consider both UGPAs and MCAT scores in context). SJTs, SEPs, and accomplishment records should be used together—as part of an “admissions toolbox”—along with data on applicants’ academic competencies, in deciding which applicants to interview.

SJT에 대한 연구가 더 필요하다. 

Future research on the use of SJTs in medical school admissions should explore different formats for 

• presenting scenarios (e.g., actors, avatars), 
• alternative response formats (e.g., rank order, narrative responses), 
• validity, and 
• the impact of coaching/faking on validity and user acceptance.

2 Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78:313–321.

22 Bardes CL, Best PC, Kremer SJ, Dienstag JL. Perspective: Medical school admissions and noncognitive testing: Some open questions. Acad Med. 2009;84:1360–1363. Acad Med. 2009;84:1360–1363. 

SJT 23 Dore KL, Reiter HI, Eva KW, et al. Extending the interview to all medical school candidates—Computer-Based Multiple Sample Evaluation of Noncognitive Skills (CMSENS). Acad Med. 2009;84(10 suppl):S9–S12. 

SJT 24 Lievens F, Sackett PR, Buyse T. The effects of response instructions on situational judgment test performance and validity in a high-stakes context. J Appl Psychol. 2009;94:1095–1101. 

SJT 25 Lievens F, Sackett PR. Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms. J Appl Psychol. 2007;92:1043–1055. 

SJT 26 Lievens F, Sackett PR. The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. J Appl Psychol. 2012;97:460–468. 

SJT 27 Ziv A, Rubin O, Moshinsky A, et al. MOR: A simulation-based assessment centre for evaluating the personal and interpersonal qualities of medical school candidates. Med Educ. 2008;42:991–998.

SEP 35 ETS. ETS Personal Potential Index. http:// www.ets.org/ppi. Accessed January 25, 2013.

Structured Interview 48 Carney A. Building a better doctor. USF Magazine. 2011:53(3). http://www.magazine. usf.edu/2011-fall/building-a-better-doctor. aspx. Accessed January 14, 2013.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

지난 25년간의 의과대학 입학과 관련된 연구와 실제(Teach Learn Med, 2013)

A Perspective on Medical School Admission Research and Practice Over the Last 25 Years

Clarence D. Kreiter and Rick D. Axelson

Department of Family Medicine, University of Iowa Carver College of Medicine, Iowa City, Iowa, USA

다양한 관점에도 불구하고 입학 프로그램은 상당히 보수적인 편이다. 실제로 지난 25년간 북미 의과대학의 학생선발 과정을 보면 거의 변한게 없다.

Despite a wide diversity of viewpoints, admission programs tend to be quiteconservative and to perpetuate the professional status quo. Infact, a comparison between current admission practices at North American medical schools and those used 25 years ago suggest that procedures have changed very little.1–4

한 가지 가능한 설명은 개념적 장벽, 그리고 기관 차원의 장벽이다.

A more likely explanation is that conceptual and organizational barriers have slowed evidence-based progress.

연구 결과 활용의 장애요인

BARRIERS TO THE USE OF RESEARCH FINDINGS

우리가 보는 관점에서, 발전을 저해하는 가장 중요한 요소는 공통의 의제와 개념 토대의 부족이다.

In our view, what has most slowed progress has been the lack of a shared agenda and conceptual foundation for guiding scholarship on admission issues.

발전을 저해하는 첫 번째 개념적 문제는 의과대학이 지원자에서 어떤 부분을 평가하고자 하는 것에 대한 모호함이다. 여러 의과대학에서 평가하는 인적 특성의 다양성은 의과대학 지원자에게 바라는 바람직한 특성에 대한 합의가 적음을 뜻한다.

The primary conceptual problem hindering admission pro- cess refinement has been ambiguity regarding what medical schools are looking for in candidates. The wide range of per- sonal qualities that are purportedly being assessed by different colleges indicate that there is little agreement regarding the desired attributes.7

AAMC의 COA는 ILWG의 결과물에 네 개의 추론과 두 개의 핵심 과학 역량을 더했다.

The AAMC’s Committee on Admissions, building upon the work of the Innovation Lab Working Group, specified four reasoning and two science core competencies 

이 역량 목록에 대한 동의를 만드는 것이 이 문제를 해결하고 지원자에게 요구되는 바람직한 특성과 관련된 여러 이슈들을 해결하는데 도움이 될 것이다.

Agreement on this list of competencies will likely be an important step toward resolution of this matter and other issues conflated with the debate over desired candidate qualities.

보다 덜한 장애요인 중 하나는 학생선발의 목적이 복잡하다는 것에 있다. 다양한 목적들이 있으며, 이는 서로 상충하기도 하고, 각각은 다시 다양한 하부 이슈로 구성되어 있어서 학생 선발에 관한 의사결정 환경은 다면적인 특성을 지닌다. 이 복잡성은 과연 validity에 대한 근거가 선발 절차에 적용되기는 하는가에 대한 의구심을 낳는다. 비록 validity theory는 한 프로그램이 어떤 목적을 가지고 있느냐에 대해서는 중립적이지만, 변화에 대한 보수적인 관점과 새로운 근거기반 방법론을 지지하는 사람간의 관점 차이는 그 사람이 psychometric validity가 특정 목적이나 목표를 향해서 편향되어있는지 여부에 어떻게 생각하느냐에 달려있다.

A less recognized obstacle to evidence-based progress relates to the complexity of admission goals. The broad and often competing concerns for social justice, educational quality, and healthcare outcomes each contain numerous subsidiary issues that ultimately create a multifaceted and complex decision environment.This complexity has led some to question whether validity evidence even applies to selection procedures. Although validity theory is neutral regarding what goals are set for a program,often the difference between a conservative viewpoint toward change and one that advocates new evidence-based methods hinges upon whether one believes that psychometric validity is inherently biased in favor of a particular goal or objective. 

근거기반 방법론을 활용하는데 관련된 또 하나의 장애는 증가하는 정치적, 법적 압박이다. 비록 학생선발 절차가 그러한 압력에 반응해야 하는 것은 맞지만, 우리는 과학적 타당도 근거가 법적/정치적 영역에서도 공정성과 효율성을 정의하는 것의 핵심이 되어야 한다고 생각한다.

A related obstacle for utilizing evidence-based methods stems from the increasingly strong political and legal pressures placed on medical schools. Although it is clear that the admissions process must respond to such pressures, we believe that scientific validity evidence should be instrumental in defining fairness and efficiency in both the legal and political arenas.

입학절차는 미국 의학교육에 큰 영향을 미쳐왔다. 효과적인 교육적 개입은 고작해야 효과크기 0.20혹은 그 이하 정도의 이득에 그치나, 근거중심 선발의 효과는 훨씬 더 강하다. 실제로 잘 설계된 선발 절차는 수행능력을 1 SD 이상 향상시킬 수 있다.

Admission procedures have had a profound impact on North American medical education. Although effective educational interventions typically produce only small gains in learning, usually with effect sizes of .20 or less, evidence-based selec- tion is comparatively far more powerful. In fact, when well de- signed, selection procedures in medical education can achieve performance gains easily exceeding 1 standard deviation.14–16

1. 인터뷰와 관련 기술들

1. The Interview and Related Techniques

어떤 도구의 타당도도 그 신뢰도의 SQRT값을 초과할 수 없기 때문에, 입학 전 면접 성적의 재현가능성은 타당도의 필요조건이다.

Because the maximum validity displayed by any measure cannot exceed the square root of its reliability, establishing the reproducibility of the preadmission interview score is a nec- essary precondition for validity.

완벽한 시험-재시험 방식의 신뢰도가 면접의 유용성을 결정하는데 가장 관련이 높기 때문에, 다면화된 일반화가능도 분석을 통해 도출한 G 계수가 - 평가자간, 평가자내 일관성 지표보다 - 가장 이론적으로 면접의 타당성을 잘 보여준다.

Because a comprehensive test–retest type of reliability is the most relevant for determining the utility of the interview, a G coefficient from a multifaceted generaliz- ability analysis, rather than an interrater or internal consistency index, has the most theoretical relevance for establishing the validity of the interview.

연구 결과를 살펴보면, 전통적인 방식의 입학면접 점수는 전반적으로 재현가능성이 매우 낮으며, 이 낮은 신뢰도로부터의 타당도 역시 면접이 중요한 성과들을 예측하는데 효과가 없음을 보여주고 있다. 이러한 결과는 입학면접을 매우 강조하는 대부분의 북미 의과대학에 시사하는 바가 크다. 명백하게, 전통적인 면접은 선발에서 중요한 요소가 되어서는 안된다. 그리고 면접 점수를 최종 결정에 활용하는 것은 공정하고 타당한 평가가 이루어질 것이라는 지원자의 기대에 반하는 것이다.

Research suggests that the overall re- producibility of the traditional preadmission interview score is very low, and the validity implication of this lowreliability have been confirmed by studies demonstrating that the interviewis in- effective in predicting important outcomes.18–22 These findings have significance for the majority of North American medical schools as they continue to maintaining a strong emphasis on the preadmission interview.1,2 Clearly, the traditional interview should not be an influential component in selection, and the use of an interview score to make the final decision may violate an applicant’s expectation of fair and valid assessment practice.9

The MMI has as its most important innovation, the use of objectivelystructuredclinical examination–style mea- surement techniques.23 This well-examined assessment method, using multiple independently rated samples of behavior, has consistently produced summary measures displaying accept- able reliability and promising validity evidence.24–28 Although the MMI has not been widely adopted, perhaps due to the com- plexity of the measurement process, it does demonstrate the fea- sibility of generating a reliable nonacademic assessment from an interview-like procedure. Although it is clear that multi- ple independent measures of performance are needed, there are competing concepts regarding which behaviors should be ob- served and rated.19,26 

2. 입학 시험

2. Admission Tests

가장 높은 예측력에도 불구하고 입학시험을 활용하는 것의 타당도에 대해서는 의견이 갈린다. 예컨대 MCAT이 어떤 임상기술이나 의과대학 후반/전공의/진료의 성취 결과를 예측하는데 비효과적이라는 결과로부터 어떤 사람들은 MCAT이 선발에서 제한적으로만 활용되어야 한다고 주장한다.

Despite generating the highest predictive coefficients, there have been conflicting views regarding the validity of using an aptitude/achievement test for selecting applicants to study medicine. For example, because the MCAT has been reported to be ineffective in predicting certain clinical skill and achieve- ment outcome measures fromthe later years of medical school, residency, and practice, some have maintained that the MCAT is of limited use in selection.31–35

메타분석을 해보면, 타당도에 대한 일반화 연구로부터 MCAT의 예측력이 비교적 의과대학 전 학년에 걸쳐서 일관되게 나타나며 MCAT이 시험점수로 나타나는 학업성취 뿐 아니라 임상기술 예측도 잘 하는 것을 보여준다.

Using meta-analytic techniques, validity generalization research has convincingly demonstrated that the predictive power of the MCAT remains relatively consistent across the medical school years and beyond and that the MCAT tends to predict clinical skills almost as well as written test-based academic achievement outcomes.15

입학시험은, 그 이름을 뭐라고 부르든, 일반적인 지능 및 추론 능력과 매우 높은 상관이 있다. 이는 입학시험을 바탕으로 한 선발은 지능을 바탕으로 한 선발이라는 의미이기도 하다.

Admission tests, whether labeled as aptitude or achievementassessments, are highly correlated with general intelligence andthe ability to reason.38,39 This implies that selection based onadmission testing also selects on general intelligence. 

다른 학생선발 맥락과 비교해보면, 의학교육에서 선발을 하는 사람들은 대른 분야에 비해서 고도로 동기부여가 되어있고 똑똑한 학생 중에서 산발하는 것이다. 이런 이유로 informed selection 방법은 겨우 의과대학 지원요건/면허취득요건을 충족한 지원자를 찾는 것보다 훨씬 더 능력 있는 지원자를 선발하는 것이다.

Compared to other selection contexts, medical education ad- mission professionals are in the enviable position of being able to select from a large pool of highly motivated and intelligent applicants. For this reason, informed selection methods are ca- pable of achieving far more than identifying applicants who will merely pass the requirements of medical school and the associated licensure and certification examinations.42

Section 4에서 논의할 것과 마찬가지로, 이 결론의 함의는 다양한 입학 프로그램의 목적을 달성하고자 할 때, 가장 타당한 모델은 MCAT 평균점수를 최대화하되, 다른 유형의 목표에 의해서 제한되도록 하는 것이다.

As we discuss in Section 4, the practical implication of this conclusion is that when attempting to achieve diverse admission programobjectives, the most valid models are those that maximize the average MCAT within the constraints imposed by other class composition goals.43,44

MCAT과 관련된 중요한 이슈 중 하나는 uGPA와 높은 상관이 있다는 점이다. 두 척도간에는 다중공선성이 높아 MCAT이 의과대학 성적이나 USMLE 점수를 예측하는데 약간정도만 기여하게 된다. 따라서 MCAT을 배제하고 uGPA만 사용하는 것이 타당해 보일 수 있으나, MCAT을 유지해야 하는 이유가 있다. 하나는 대학 성적의 의미는 대학이 어디인지 전공이 무엇인지 따라서 매우 다르다는 것이다. 또한 이러한 이유로 uGPA만 활용하는 것은 불공정하고 교육 절차를 훼손시킬 수 있다. MCAT은 uGPA를 여러 대학간 동질화시키는 효과적인 수단이다.

An additional and important interpretive issue concerning MCAT research relates to the fact that admission test scores correlate highly with undergraduate grade point average (uGPA). Multicollinearity between the two measures results in the MCAT contributing only modestly to the incremental vari- ance explained in medical school grades and USMLE scores.14 Althoughit might therefore seemreasonable tosimplyuse uGPAwithout MCAT, there are compelling reasons to retain MCATin the selection process. One of the primary shortcomings ofcollege grades relates to the fact that the meaning of uGPAvaries dramatically across undergraduate institutions and ma-jors.45 The variability in standards across institutions suggeststhat it is unfair and likely damaging to the educational process,to use uGPA in isolation. Because MCAT is an effective meansof equating uGPA across institutions, it also serves a vital rolein maintaining the integrity of our educational metrics.46 

3. 개인 역량에 대한 다른 척도들

3. Other Measures of Personal Competencies

Although their average predictive validity appears to be quite low, Tett and Christiansen50 argued that when well-constructed personality tests are used under the proper conditions, they can achieve useful levels of predictive validity. For our purposes, the issue then becomes whether the proper test(s) and conditions can be developed for selecting medical school applicants with sufficient predictive validity. 

There is considerable skepticism regarding whether theseconditions can be attained in medical student selection pro-cesses. As Bardes et al.52 noted, although there is some ev-idence of links between noncognitive measures and medicalstudent performance, these studies were based on medical stu-dents rather than applicants. Given the high stakes nature of theselection process, there is strong incentive for applicants to givethe answers desired by an admissions committee rather than an-swering questions candidly

자기소개서는 세 번째 방법

Autobiographical essays are a third method for collective information about nonacademic attributes. 

의과대학이 이 정보를 수집하고 점수를 매기는 방법은 매우 다양하며 멕마스터대학에서의 연구에 따르면 평가자간 신뢰도가 낮아서 성과 척도와 낮은 상관관계만을 보여준다. 

There appears to be much variation in how medical schools collect and score this information. Re- search conducted at McMaster University suggests that scores on the essays have little correlation with outcome measures, at least in part, because of their low interrater reliability.22 Subse- quent research found that the reliability and predictive potential for these scores was improved by administering the essays in an onsite, proctored, time-controlled environment and by rating the resulting essays using a horizontal scoring method.53

따라서 위의 세 가지 중에서는 현재로는 지침에 따라 작성하는 표준화된 추천서가 personal competency를 평가하는 가장 좋은 방법으로보인다.

Thus at present, standardized letters of recommendation guided by instructions such as those developed by the AAMC’s Admission Initiative appear to be the most promising of these three methods for gathering reliable and valid information about candidates’ personal competencies through admission processes.

SJT와 Sternberg's successful intelligence test도 있다.

Situational judgment tests (mentioned earlier) and Sternberg’s successful intelligence test54 are two examples of ability-based tests that might capture additional information about applicants’ personal competencies. Successful intelli- gence includes creative and practical skills that enable individuals to envision, evaluate, and implement practical solutions to problems. Using a sample of 793 college students, Sternberg54 showed that such measures substantially improved the predic- tion of students’ college GPA.

4. 합격자 결정 과정

4. The Decision Process

안타깝게도, 지난 25년간 성과 척도를 개발하는데 거의 자원이 투입되지 않았다. 많은 연구들이 성공적인 학생 똔느 의사를 만드는데는 많은 요소가 필요함을 인정하지만, 정확상과 편의성을 이유로 성적/면허시험점수/증명서 등에 안주하고 있다.

Regrettably, over the last 25 years, there have been fewresourcesallocated toward the development of outcome measures. Moststudies acknowledge that there are many elements that define a successful student or physician, but for reasons of measurement precision and convenience, settle for grades, licensure scores, and/or successful certification.

사용가능한 다양한, 하지만 불완전한 지표들을 합하여 척도를 만드는 것이 지금까지 간과되어 온 방법이다. 통계적인 insight에 따르면, 척도가 다양하더라도 우리가 성공적인 의사라고 정의한 그 방향을 모두 일관되게 향하고 있다면 그것들을 합함으로써 신뢰도와 타당도 높은 결과를 얻을 수있다.

One overlooked method for building an index that could pos- sibly reflect physician success might be found in a composite measure constructed from a large set of available but imperfect indicators. Statistical insights suggest that if the components of a composite measure all vary in the same direction as what we are trying to define/predict (a successful physician), a reli- able and valid composite is likely to evolve.

어떤 사람은 어떻게 이렇게 다양한 요소들을 - 각각이 기여하는 정도가 어느 정도인지도 모르고서 - 합할 수 있느냐고 무렁볼 수 있는데, 통계 연구를 살펴보면, 비록 어떤 요소들이 더 중요하고 더 신뢰도가 높을 수있지만, 표준화된 점수를 합하는 것 만으로도 가중치가 적용된 점수와 거의 완벽하게 상관관계를 가진다.

One might wonder howweights for such diverse components could possibly be generated when we have so little knowledge about each indicator’s relative contribution in defining the con- struct. Fortunately, statistical research demonstrates that this problem is easily solved. Although it is reasonable to regard some of these component indicators as more important and re- liable than others, statistical evidence clearly indicates that a sum of standardized scores (simple unit weighting) will gener- ate a composite almost perfectly correlated with any optimally weighted composite we might derive with more complete information.60

19. Axelson RD, Kreiter CD. Rater and occasion impacts on expected pread- mission interview reliability. Medical Education 2009;43:1198–202.

26. Ziv A, Rubin O, Moshinsky A, Gafni N, Kotler M, Dagan Y, et al. MOR: A simulation-based assessment centre for evaluating the personal and interpersonal qualities of medical school candidates. Medical Education 2008;42:991–8.

43. Kreiter CD. The use of constrained optimization to facilitate admission decisions. Academic Medicine 2002;77:148–51. 44. Kreiter CD, Stansfield B, James PA, Solow C. A model for diversity in admissions: Areviewof issues and methods and an experimental approach. Teaching and Learning in Medicine 2003;15:116–22.

48. Siu E, Reiter H. Overview: What’s worked and what hasn’t as a guide to- wards predictive admissions tool development. Advances inHealthSciences Education 2009;14:759–75.

49. Association of American Medical Colleges. Guideline for writing letter of evaluation for a medical school applicant. 2011. Available at: https://www. aamc.org/initiatives/admissionsinitiative/letters./ Accessed July 15, 2013.

A perspective on medical school admission research and practice over the last 25 years.

학부의학교육과 의사 프로페셔널리즘의 토대(JAMA, 2015)

Undergraduate Medical Education and the Foundation of Physician Professionalism

Darrell G. Kirch,MD Association of American Medical Colleges,Washington, DC.

Maryellen E. Gusic, MD Association of American Medical Colleges,Washington, DC.

CoriAst,MHSA Association of American Medical Colleges,Washington, DC.

프로페셔널리즘은 이렇게 정의된다.

Professionalism is the demonstrated “commitment to carrying out professional responsibilities and an adherence to ethical principles”1 

여러 전문직 조직에서 어떻게 의사들의 프로페셔널리즘을 유지할 것인가에 대한 논쟁이 있지만, 학부의학교육에서는 공유거버넌스 모델(shared governance model)이 이 중요한 역량 개발의 프레임워크를 제공한다.

Although there is current controversy regarding how diverse professional organizations should ensure professionalism among practicing physicians, during undergraduate medical education a shared governance model, as described below, provides the framework for developing and accessing this critical competency.

프로페셔널리즘의 토대는 의과대학 이전에 시작된다.

The Foundation of Professionalism Begins Before Medical School

Aspiring physicians, through many years of personal ex- periences prior tomedical school, establishthe “prepro- fessional” foundation for competence in professional- ism, making it important to assess preprofessional attributes in medical school admissions.

전-프로페셔널리즘 역량에 대한 평가는 의과대학에서 "전인 평가"의 핵심이다.

The evaluation of preprofessional competencies is central to the “holistic review” of medical school applicants.

의과대학 학생선발 절차의 중심에는 MCAT이 있는데, 이 MCAT은 최근 21세기 의사에게 필요한 더 광범위한 포트폴리오를 요구하는 방향으로 수정되었다.

Central to the admissions process, the Medical College Admissions Test (MCAT) has recently been revised to emphasize the broader portfolio of skills required by physicians in the 21st century. 

개정된 MCAT은 다른 몇 가지 선발절차의 변화와 동반되고 있는데, 그 중 하나는 추천서 작성에 대한 가이드라인을 만든 것이다. 또한 표준화된 지원서 양식을 통해 학사 학업내용 뿐 아니라 관련된 개인 경험을 기술하도록 하였다.

The revised MCAT has been coupled with other changes in the admissions process, including the creation of guidelines for letters of recommenda- tion to ensure inclusion of information about the core competencies for entering medical students and, in the standardized application form, asking applicants to document relevant personal experiences in addition to their coursework.

도입 초기이지만 MMI는 의과대학 수행능력을 예측하는 것으로 보인다.

In early stages of implemen- tation, MMIs appear predictive of future performance in medical school.4

아직 미국 내에서는 검토단계이지만 SJT는 벨기에 의대생 선발에서 1997년부터 사용되어왔다.

Although still under assessment in the United States, an SJT has been used by Belgium for medical school admission since 1997.2

Increasingly, it is recognized that professionalism is not cultivated solely by role-modeling in clinical settings but rather that pro-fessionalism must be taught early, longitudinally, and deliberately using both targeted instruction and experiential learning. The LCME accreditation standards formalize the institutional responsibility by requiring that medical schools maintain a learning environment that cultivates the development of professionalism among learners. 

중요한 점은, 한 학생이 전문직으로 성장하는 것에 대한 책임은 행정, 교수, 학생이 모두 공유하는 것이다.

Importantly, this responsibility for students’ professional development is shared with administrators, faculty, and students serving on various administrative committees.

To support these assessments, more than half of medical schools in the United States and Canada rely on 

“defined, written standards of non-cognitive behavior, [including] 

• honesty; 
• professional behavior; 
• dedication to learning; 
• professional appearance; 
• respect for law and others; 
• [and adherence to standards related to] confidentiality; and 
• [lack of issues related to] substance abuse,” 

in addition to the academic standards for promotion established by each school.6 

프로페셔널리즘 평가는 생화학 지식 평가보다는 덜 정확할지 모르나, 다양한 도구들이 있다.

Although assessing competence in professionalism may be less precise than assessing competence in the knowledge of biochemistry, today there are tools to assess professionalism in students, including 

• patient evaluations, 
• self and peer assessments, 
• behavioral observation, 
• psychological testing, and 
• even structured examinations.6

"한 아이를 키우는데는 한 마을이 필요하다"라는 말이 있듯, 이는 의사를 양성하는 것에도 마찬가지이다. 

The dictumthat “it takes a village to raise a child” also appears to be true of educating a physician. Although the ultimate responsibility for professionalism rests with the physician aspirant, multiple parties are involved in shaping the preprofessional attributes of aspiring physicians, as well as those in the undergraduate process of training, including 

committees for admissions, curriculum, and progression that engage administrators, faculty, and students in the oversight, development, and assessment of professionalism. Affiliated organizations, including the LCME and AAMC, play a significant role in setting standards and providing tools related to teaching and assessing professional development.

4. Pau A, JeevaratnamK, Chen YS, Fall AA, Khoo C, Nadarajah VD. The Multiple Mini-Interview(MMI) for student selection in health professions training—a systematic review. Med Teach. 2013;35 (12):1027-1041.

6. Boon K, Turner J. Ethical and professional conduct of medical students: review of current assessment measures and controversies. J Med Ethics. 2004;30(2):221-226.

Undergraduate medical education and the foundation of physician professionalism.

미래 의사 선발: 미래 보건의료인력의 핵심(Acad Med, 2013)

Selecting Tomorrow’s Physicians: The Key to the Future Health Care Workforce

Kelly E. Mahon, MA, Mackenzie K. Henderson, and Darrell G. Kirch, MD

최극 미국 내 의료혁신은 세 가지 목적을 가지고 있다. 하나는 개개인의 건강을 향상시키는 것, 둘째는 인구집단의 건강을 향상시키는 것, 그리고 마지막으로 비용을 낮추는 것이다. 이 목적을 달성하기 위해서 전통적으로 '자율성'을 바탕으로 진료를 해왔던 의사들은 점차 팀-기반 진료 모델의 구성원이 될 것을 요구받고 있다.

Recent health care reform efforts in the United States have focused on the “triple aim”1 of improving health care for individuals, improving population health, and lowering costs. Physicians, who traditionally have practiced with considerable autonomy, will be required to become members of the team-based patient care models that are necessary to achieve these goals

의과대학 입학: 역사적 유물

Medical School Admissions: A Historical Legacy 

플렉스너가 20세기 초반, 북미 전역에 걸쳐 의과대학을 평가했을 때, 의학전교육요건(premedical education requirements)과 의과대학 입학절차가 부실하다는 점을 지적했다. 이는 미국에 기준에 미달하는 의과대학들이 횡행하는 이유이며 "의학교육과 의료에 전례없는 해를 끼칠 수 있는 토양"이라고 묘사했다.

When Flexner traveled across North America in the early 20th century, he decried the lack of rigor in premedical education requirements and medical school admission processes, describing the proliferation of substandard medical schools in the United States as “the fertile source of unforeseen harm to medical education and to medical practice.”3 

플렉스너 보고서의 중요한 유산 중 하나는, 미래 의사가 최소한의 기초과학 지식을 갖춰야 한다는 것이다.

The key enduring legacy of the Flexner Report is its argument that future physicians should possess a minimum threshold of knowledge in the basic and natural sciences.4 

MCAT시험은 의과대학지원자의 과학지식에 대한 성취 정도를 평가할 뿐 아니라, GPA와 더불어 의과대학에서의 성적, 의사면허시험에서의 성적을 예측하는 도구로 활용되어 왔다.

The MCAT exam has become the tool of choice not only to measure medical school applicants’ mastery of scientific content, in conjunction with their grade point averages, but also to act as a reliable predictor of success in medical school and initial licensure examinations.5 

오랫동안 의과대학 지원자를 GPA나 MCAT 점수를 넘어선 어떤 기준에 의해서 평가해야 한다는 열망이 있었으며, 1980년대 초반에는 의과대학 지원자를 단순한 예비 학자가 아니라 하나의 인간으로서 평가하는 live interview가 등장하였다. 최근 면접이 개선됨에 따라 면접이 스크리닝 도구로서 강점을 가지게 되었음에도 불구하고, 면접은 전통적으로 약하고, 주관적이고, 비일관된 도구라고 인식되어 왔다.

There long has existed a clear need to assess applicants beyond their grades and MCAT scores. By the early 1980s, live interviews emerged as a tool to help admissions officers get to know an applicant as a person and not merely as a scholar.6 Although recent innovations, as we will discuss below, are showing great promise to strengthen the interview as a screening tool,7 interviews traditionally have been a relatively weak, subjective, and inconsistent means by which to assess medical school applicants.8

미래 의료인력 선발: 의료 혁신의 핵심

Selecting the Future Physician Workforce: A Key to Health Care Reform

플렉스너의 연구 결과에 따라서 만들어진 입학 시스템이 20세기와 21세기 의사들이 전통적인 자연과학 분야의 튼튼한 토대를 갖출 수 있도록 성공을 거둬온 것은 사실이다. 그러나 이는 역설적으로 보건의료 시스템을 변혁할 수 있는 혁신적 의사를 찾아내는데는 약점을 보여왔다.

Although the admission system created in response to Flexner’s findings has been successful in ensuring that 20th- and 21st-century physicians are grounded in the natural and traditional life sciences, it has fallen short in identifying the innovative physicians who can transform the health care system.

의과대학 입학에 대해 생각할 때 몇 가지 함께 고려할 요인이 있다.

Several major factors have converged to influence thinking about medical school admissions. 

• national debate surrounding health care reform 
• passage of the Patient Protection Affordable Care Act— 
• issues regarding professionalism 

전인적 학생선발

Supporting Holistic Admissions

Holistic admissions, an integral component of the AI, refers to a “flexible, highly individualized process by which balanced consideration is given to the multiple ways in which applicants may prepare for and succeed as medical students and doctors.”24

This process complies with the U.S. Supreme Court’s “holistic review” rubric, which was established in 2003 by Grutter v. Bollinger, and calls for an individualized review of each applicant that considers how that applicant might contribute to a diverse educational environment. Evaluation criteria for a holistic review process must be mission driven, broad based, institution-specific, and applied across the applicant pool consistently.25 Holistic review has three goals: 

• to assess applicants’ academic readiness for medical school, 
• to identify and assess applicants’ interpersonal and intrapersonal competencies, and 
• to encourage diversity in medical education.

학업 준비도 재정의

Redefining academic readiness

의학교육에서의 역량 정의

To define medical education competencies, two working groups identified the skills and knowledge that future physicians should possess on entry to or completion of medical school. 

• Issued in 2009, “Scientific Foundations for Future Physicians” 
• The companion report, “Behavioral and Social Science Foundations for Future Physicians,”

최근 AAMC와 다른 다섯개의 협회가 IPEC를 구성

More recently, the AAMC and five other health associations representing schools of osteopathic medicine, dentistry, nursing, pharmacy, and public health jointly created the Interprofessional Education Collaborative (IPEC).

여기서는 전문가-간-역량을 정의하고 보고서 발간

This group initially defined four interprofessional competencies that health professions students should acquire over the course of their training: 

• values and ethics, 
• understanding roles and responsibilities, 
• interprofessional communication, and 
• teamwork. 

The result of IPEC’s efforts, “Core Competencies for Interprofessional Collaborative Practice,” represents the first time consensus has been reached about competencies required for team- based practice in a variety of settings, including in the clinic and at the bedside.27

MCAT이 현재 의과대학생 선발에서 중요한 도구로 사용되고 있기에 MR5는 2015년부터 MCAT의 개선된 버전을 사용할 것을 권고하였다. 가장 큰 변화는 행동과학, 사회과학 개념이 포함된다는 것이다.

In recognition of the MCAT exam’s status as an important tool for medical student selection,2 the fifth MCAT review (MR5) committee recommended, and the AAMC Board of Directors approved in February 2012, revisions to the MCAT exam beginning in 2015.19,28–30 One of the most prominent changes is that the 2015 exam will add a section that tests knowledge of concepts from the behavioral and social sciences to complement testing in the basic and natural sciences.31

내적 역량과 대인관계 역량 도출 및 평가

Identifying and assessing interpersonal and intrapersonal competencie

전인적 평가의 두 번째 목표는 전인격을 갖춘 의사가 될 학생을 찾아내는 것이다. AAMC는 바람직한 역량을 아래와 같이 도출했다.

Holistic review’s second goal is to identify applicants who possess the traits, experiences, and attributes that will lead them to become well-rounded physicians. In 2013, the AAMC identified the most desirable interpersonal and intrapersonal competencies for entering medical students34,35 (see Table 1).

두 번째는 SJT를 활용하여 평가하는 것이다. SJT는 다음과 같은 것이다.

Secondly, the AAMC is exploring the development of a situational judgment test (SJT) as another tool to probe applicants’ interpersonal and intrapersonal competencies.39 SJTs, which “confront applicants with written or video-based scenarios and ask them to indicate how they would react by choosing an alternative from a list of responses,” have shown great promise in identifying interpersonal skills.40

앞에서 말한 바와 같이 새로운 면접 기술이 등장하고 있으며 MMI는 멕마스터 의과대학에서 처음 도입되어 대부분의 캐나다 의과대학과 미국 의과대학 중 22개 이상의 의과대학에서 활용중이다.

As mentioned earlier, new interview techniques are emerging to allow medical schools to probe better dimensions of applicants’ competencies, ranging from how applicants respond to novel situations to their reactions to an ethical conflict. The multiple mini-interview (MMI) was pioneered by the Michael DeGroote School of Medicine at McMaster University and is now employed by the majority of Canadian medical schools and more than 22 U.S. medical schools.7,39

의학교육의 다양성 지탱하기

Supporting diversity in medical education

이러한 변화는 미래의 의사들이 높은 수준의 문화적 역량을 갖출 것을 요구할 것이며 이는 다음과 같이 정의된다.

This change will require that tomorrow’s physicians possess a high degree of cultural competence, which has been defined as “a set of congruent behaviors, knowledge, attitudes, and policies that come together in a system, organization, or among professionals that enables effective work in cross-cultural situations.”43

Page는 다양한 배경에서 온 다양한 사람으로 이뤄진 그룹이 문제해결을 더 잘 잘하며, 어떤 개개인보다도 뛰어나다는 것을 보여주었다. 또한 추가적으로 의과대학생들은 같은 의대생들의 다양성을 가치롭게 생각하며, 학업경험은 물론 환자를 보는 경험 역시 이 다양성에 의해서 향상된다고 평가했다.

Page44 has shown that diverse groups of people from varied backgrounds do better at problem solving and, in many ways, are smarter than any individual. Further evidence shows that “students in medical schools value diversity in their classmates and find both the academic experiences and their abilities to work with patients from differing backgrounds enhanced by this diversity.”45

100년 전, 의과대학은 표준화된 시험을 통해서 과학적 배경이 잘 갖춰진 학생을 선발하였으며, 이것이 플렉스너 보고서에서 드러난 문제를 해결하는 확실한 방법이었다. 그러나 100년이 지난 지금, 환경은 변하고 있다.

A century ago, the academic medicine community concluded that providing physicians with a rich scientific background, verified through the use of standardized tests, was the definitive answer to addressing the problems revealed by the Flexner Report. As the last 100 years have demonstrated, however, changing circumstances in the health care landscape necessitate constant transformation.

23 Swick HM. Toward a normative definition of medical professionalism. Acad Med. 2000;75:612–616. 

24 Addams AN, Bletzinger RB, Sondheimer HM, White SE, Johnson LM. Roadmap to Diversity: Integrating Holistic Review Practices Into Medical School Admission Processes. Washington, DC: Association of American Medical Colleges; 2010. https://members.aamc.org/eweb/upload/ Roadmap%20to%20Diversity%20 Integrating%20Holistic%20Review.pdf. Accessed August 21, 2013.

25 Witzburg R, Sondheimer H. Holistic review: Shaping the profession of medicine one applicant at a time. N Engl J Med. 2013;368:1565–1567. http://www.nejm.org/ doi/pdf/10.1056/NEJMp1300411. Accessed August 21, 2013.

38 Association of American Medical Colleges. Letters of evaluation guidelines. https://www.aamc.org/ initiatives/admissionsinitiative/332572/ lettersofevaluationguidelines.html. Accessed August 21, 2013.

27 Interprofessional Education Collaborative Expert Panel. Core Competencies for Interprofessional Collaborative Practice: Report of an Expert Panel. Washington, DC: Interprofessional Education Collaborative Expert Panel; 2011. http://www.aacn.nche. edu/education-resources/ipecreport.pdf. Accessed August 21, 2013.

Selecting tomorrow's physicians: the key to the future health care workforce.

의과대학 입학시 스무고개 수행능력을 통한 임상수행능력 예측 (Med Educ, 2015)

Twenty Questions game performance on medical school entrance predicts clinical performance

Reed G Williams1 & Debra L Klamen2

"지식이 있다"는 것은 무엇인가? - 네 가지가 있음

-'누가'에 대한 지식

-'무엇을'에 대한 지식

-'어떻게'에 대한 지식

-'언제'에 대한 지식

White,2 in a careful analysis of what it means to possess knowledge, concluded that knowledge is manifested as ability and that ability takes many forms. The more typical and easily measured forms of knowledge involve knowing...

• who (e.g. who invented the transistor) or 
• what (e.g. what significant event occurred in New York City on 11 September 2001). 

However, knowing also includes knowing 

• how (e.g. how to ride a bike) and 
• when (e.g. when to add elements to a mixture in a process). 

'안다'라는 것은 단순히 기존에 알려진 답을 다시 되풀이하는 것이 아니라 새로운 문제에 대한 답을 찾아내는 기능을 뜻한다.

Most importantly, White argued that knowing does not merely involve the ability to produce old answers previously acquired, but also the facility to find new answers to new problems.2

기존의 지식은 최소한 다섯 가지 형태로 결합될 수 있다. (사실, 개념과 원리, 절차, 전략, 신념)

To cope successfully with the tasks and demands of everyday life, humans must be proficient in combining previously learned knowledge, skills and attitudes (beliefs) into at least five forms: 

• facts (bits of information); 
• concepts and principles (e.g. knowledge of cause-and-effect relationships); 
• procedures (e.g. knowledge of step-by-step processes regarding how and when to carry out an action, such as in how and when to carry out long division computations); 
• strategies (general methods for approaching problems such as by breaking a problem into parts), and 
• beliefs (e.g. stable attitudes that lead to predictable behaviours, such as beliefs about the factors that lead to patient behaviour changes). 

이 다섯 가지 요소의 다양한 조합을 통해서 업무를 수행하게 된다.

All five fit White’s2 construction of knowledge as ability. Various combinations of these five elements are compiled and drawn upon to meet the various tasks and demands placed on people.

본 연구는 스무고개 게임의 능력이 일상생활에서 얻은 지식과 어떻게 그것을 구조화하고, 효과적으로 저장하여 효율적/효과적으로 인출, 결합, 활용할 수 있는가를 보여줄 수 있다는 전제에서 시작하였다.

The present study is based on the premise that the TQ parlour game tests the knowledge people have acquired in the course of their everyday lives and how well their organising and storing of that knowledge allows them to efficiently and effectively retrieve, combine and use it to address the challenges posed in everyday life.

본 연구에서 가능한 추가적 이점은 '인내심'의 척도로 활용가능하다는 것이다. 답을 맞추지 못하였더라도 20개의 질문을 모두 한 학생이 있고 중간에 포기한 학생이 있다.

A further benefit of this task is that it provides a  measure of perseverance. Students who fail to solve the problem posed and quit before they have used their entire quota of questions may be providing evidence of low perseverance, which may be a negative indicator of potential success as a medical student.

Study design

This was a prospective, longitudinal, observational cohort study. All students entering Southern Illinois University School of Medicine in 2009 were invited to play a single game of TQ on a non-medical topic during the first week of medical school

Description of TQ tasks and game process

Each participating entering student played a single game of TQ in a one-to-one encounter with the investigator at the time of his or her orientation to medical school. The TQ tasks posed were based on non-medical knowledge acquired through normal life experiences. A number of objects (correct answers) were selected in advance. The object (correct answer) for each participant was selected using a random selection process.

The investigator kept an essentially verbatim record of the number and nature of the questions asked and guesses offered by the student. The investigator also kept detailed notes about the strategies used in playing the game.

질문의 접근법을 네 가지로 구분

Based on the notes taken about student performance on the TQ task, a second investigator, blinded to SCCX and diagnosis justification (DJ) performance, classified the performance into one of four groups based on the student’s approach to the task: Essentially random; Somewhat random; Somewhat logical, and Logical.

논리적 접근은 이런 것이다.

These performances: 

(i) started out with broad questions in order to define the right path; 

(ii) built new questions based on previous answers, and 

(iii) offered educated guesses rather than random (off-the-wall) guesses.

Diagnosis justification exercise (진단 정당화 시험)

For eight cases, students were required to provide a written justification for their final diagnosis as part of the post-encounter exercise. The specific instructions given to students were as follows: ‘Please explain your thought processes in getting to your final diagnosis; how you used the data you collected from the patient and from laboratory work to move from your initial differential diagnoses to your final diagnosis. Be thorough in listing your key findings (both pertinent positives and negatives) and explaining how they influenced your thinking’.

Each response was blindly read and rated by two physician judges (the case author and one additional physician who was an expert in diagnostic reasoning). More details on this task and its use have been published in earlier manuscripts.11,12

Logical Group이 하위 두 개 그룹보다 SCCX와 DJ Performance가 우수함

Logical Group이 SCCX와 DJ에서 가장 우수하며 경향성을 보임

답이 틀렸어도 끝까지 20문항을 모두 질문했는지 여부에 따른 차이

DJ exercise 예측력에 있어 MCAT 점수와의 비교

Effect size interpretations (large and medium) are based on conventions described by Kirk.13

다른 의과대학, 문화권에서 확인이 필요함

As with any research results, confidence in these results will increase if the study can be successfully replicated in this and other medical schools. It is especially important to determine whether similar results are observed in other cultures because it is certainly possible, if not probable, that the results will vary based on the child-rearing and educational practices observed elsewhere.

Twenty Questions game performance on medical school entrance predicts clinical performance.

입학 후 학업능력 예측: 학생의 배경환경과 과거 학업능력의 상대적 중요성 (Med Educ, 2015)

Predicting performance: relative importance of students’ background and past performance

Karen M Stegers-Jager,1 Axel P N Themmen,1,2 Janke Cohen-Schotanus3 & Ewout W Steyerberg4

academic failure와 관계되어있다고 보고된 입학전 특성들

Pre-admission characteristics that have been reported to relate to academic failure are ethnic minority status,6,7 maturity,8,9 male gender,7,8,10 and lower levels of previous academic performance, in particular low Medical College Admission Test (MCAT) scores and low science grade point averages (GPAs).7–9

의과대학 입학 후 첫 달의 성적과의 관계를 밝힌 것도 있음.

Several studies have confirmed the relationship between student performance during the first months at university and subsequent performance. 15–17 A recent study by Winston et al. showed that results on an examination administered after the first 2 weeks of medical school represented a strong early predictor of success or failure.4

그러나 각 의학교육의 단계마다 낮은 학업능력의 risk factor는 서로 다름

In addition, it has been shown that risk factors for poor performance vary at different stages of the medical course.7,19,20

Participants and procedure

코호트 선택의 이유

Students from six consecutive cohorts (2002–2007) at Erasmus MC Medical School were included in this study (n = 2357). We selected these six cohorts for two reasons: (i) the curriculum did not change during this period, and (ii) data on the ethnicity of these cohorts were available from a national database of students in higher education in the Netherlands 1cijferHO. Data on academic performance were derived from the university student administration system and confidentiality was guaranteed. Because data were collected as part of regular academic activities, individual consent was not necessary.

모든 variable을 넣고 logistic regression.

As all variables are known to be associated with medical school performance, all were entered simultaneously in a multivariable logistic regression model. We also included cohort as a stratification variable. Statistical interaction terms were used to study the potentially differential effects of one predictor by values of another predictor.

1학년의 첫 4개월의 GPA의 영향은 남학생보다 여학생에서 두드러졌다. 같은 성적이라면 남학생이 1학년을 이수할 가능성이 더 높았다.

The effect of GPA at 4 months on the Year 1 completion rate was less prominent for males than for females: male students with a high GPA at 4 months were less likely to complete the Year 1 course on time than female students with a similar GPA, whereas male students with a low GPA at 4 months (< 4.5 on a scale of 1–10) were more likely to complete the Year 1 course on time than female students with a similar GPA.

본 연구는 미래 학업능력 예측에 가장 최근의 학업능력(입학 전이든 후든)의 중요성을 보여준다. 그러나 임상실습 단계에서는 학생의 background가 주요한 예측인자로 작용한다.

This study confirms the importance of the most recent past performance – either before or at medical school – as a predictor of future performance in pre-clinical training. However, it also reveals that in clinical training the student’s background becomes the main predictor of performance as students from all minority groups and first-generation university students had a higher risk of achieving lower clinical grades in a model that included pre-clinical performance.

입학전 GPA는 그 자체로는 강력한 예측인자였지만 입학 후 Performance가 포함된 모델에서는 유의성을 잃었다. 이 말은 의과대학 입학 직후 4개월간의 학업능력이 훨씬 더 중요하다는 것을 보여준다. 의과대학에서의 첫 시험에서의 성적을 바탕으로 위험학생을 선별해내는 것을 권고하는 것이며, 학생과 학업환경의 상호작용에 따른 결과가 입학시점에서의 자질 평가를 바탕으로 한 것보다 더 정확하다는 것을 보여준다. 

However, our study offers a more nuanced picture. Although pu-GPA was the most important predictor in the model that included only preadmission data, this factor was rendered insignificant by the addition of early performance at medical school. In other words, the factor pu-GPA was greatly outweighed by study performance data that became available during the first 4 months at medical school. This confirms the suggestion that the identification of at-risk students based on the first results of the interaction between a student and the academic environment is more accurate than identification using entry qualifications and is also in line with the findings of others.4,18

그러나 남학생의 경우는 1학년 학업능력을 첫 몇 개월의 성적으로 예측하는 것에서 예외였는데, 여학생에서보다 남학생에서 예측력이 더 낮았다. 이는 아마도 남학생의 self-efficacy가 더 높은 것이 원인일 수 있다. 남학생은 첫 4개월동안 성적이 높으면 스스로를 과대평가 할 수도 있으나, 낮은 성적을 받는 것 역시 여학생에 비해서 self-confidence에 덜 detrimental하다.

Apparently, the differences found in Year 1 performance for these subgroups can be explained to a large extent by performance during the first months at medical school. An exception to this is male gender, as being male remains a predictor of poorer Year 1 performance after the addition of early medical school performance. The interaction effect we found between gender and GPA at 4 months suggests that early performance is less predictive of later performance for males than it is for females. This may be explained by higher self-efficacy in males than in females, which has been reported previously in medical students.30,31 In male students, high grades during the first 4 months may lead to an over-estimation of their own ability, whereas achieving low early grades may be less detrimental to their self-confidence than it is in female students

인종과 사회적 배경은 임상 수행능력의 중요한 예측인자였다. 이는 임상수련은 전임상교육과는 서로 다른 메커니즘에 의해서 작동됨을 보여준다. 한 가지 가능한 이론은 문화적 자본(cultural capital)이다. Bourdieu에 따르면 이는 다음과 같이 정의된다. 비전통적인 소수인종 혹은 가계 내 최초대학입학자의 경우 이들의 cultural capital이 institutional habitus에 잘 맞지 않는다고 볼 수 있다. 임상실습에서는 보다 주관적인 평가가 많이 작동하기 때문에, 문화적 자본이 전임상교육 기간보다 더 강하게 작동할 수 있다. 

The finding that ethnicity and social background were important predictors of clinical performance, even after adjusting for pre-clinical performance, suggests that performance in clinical training is explained by mechanisms other than those referred to in pre-clinical training. A possible mechanism refers to the concept of cultural capital, which, according to Bourdieu, can be understood as ‘knowledge of the norms, styles, conventions and tastes that pervade specific social settings and allow individuals to navigate them in ways that increase their odds of success’ (see Massey et al.33). The cultural capital of non-traditional – ethnic minority and first-generation university – medical students is less likely than that of traditional medical students to be recognised and positively valued within medical school; that is, it does not fit the ‘institutional habitus’ (see Thomas34). As more subjective examination methods are used in clinical training than in pre-clinical training,35 it may be that the role of cultural capital is more prominent during clinical than pre-clinical training. Although further research is required to confirm these proposed effects of cultural capital, our assumption is supported by our finding that having a medical doctor as a parent is related to poorer performance in pre-clinical but better performance in clinical training.

임상실습과 관련하여 또 다른 가능성은, 문화적 자본과도 관련되어있지만, 문화적 편견이 평가자에게 작용했을 수 있다는 것이다. 사람들은 자기와 같은 그룹에 속한 사람을 보다 신뢰하는 경향(in-group bias)가 있고, 자신과 비슷하거나 그들이 좋아하는 사람과 비슷한 경우 더 신뢰하는 경향이 있다(similarity principle) 이러한 것을 평가자가 스스로 통제하려 하지만, 소수인종이나 first-generation university student는 "전통적" 학생그룹보다 낮은 점수를 받았을  수 있다.

Another possible mechanism in clinical training, which is related to cultural capital, is cultural bias on the part of the examiners. Inevitably, people will have more positive views of those they believe to be part of their group (referred to as ‘in-group bias’36) and people tend to trust those who are similar to themselves or who are similar to people they like (a phenomenon known as the ‘similarity principle’37). Unless traditional examiners are aware of and attempt to control these automatic reactions,38 it is likely that ethnic minority and first-generation university students will receive lower grades than their traditional counterparts. More detailed experimental studies may assist in elucidating the processes underlying judgement and decision making in clinical assessments.

본 연구의 첫 번째 한계점은 일부 factor에 대해서 제한된 숫자의 학생만 응답한 것인데 missing value에 대해서 multiple imputation 기법을 활용하였다. 

A first limitation of our study is that data on the pre-admission factors ‘first-generation university student’ and ‘medical doctor as parent’ were collected for a restricted number of participants. However, to deal with the missing values, we used the technique of multiple imputation, which is widely accepted as suitable.26 As they allow the use of data that are available for other predictors that would otherwise be lost, imputation methods, especially multiple imputations, are superior to complete case analysis. 26,41,42 The ORs calculated in the imputed dataset in our study were similar and, if different, generally more conservative than the ORs in the unimputed dataset (Table S1).

Predicting performance: relative importance of students' background and past performance.

MMI로 평가한 인적특성의 변동요인(Medical Teacher, 2014)

Variance in attributes assessed by the multiple mini-interview

NIKKI BIBLER ZAIDI1, CHRISTOPHER SWOBODA2, LEIGH LIHSHING WANG2 & R. STEPHEN MANUEL3

1University of Michigan Medical School, USA, 2University of Cincinnati, USA, 3University of Cincinnati College of Medicine, USA

Introduction

가장 신뢰도가 높은 입학면접은 어떤 형태일 것인가?, MMI에 대한 논의. MMI의 다면표집법은 낮은 신뢰도를 극복하는 수단으로 긍정적 평가를 받았다.

The medical school preadmission interview (MSPI) remains a widely used tool in medical school admissions (Monroe et al. 2013); therefore, discussions regarding the most reliable and valid MSPI format continue to evolve (Edwards et al. 1990; Goho & Blackman 2006). Over the past decade, the multiple mini-interview (MMI) has gained considerable attention as an alternative to more traditional MSPI formats. The MMI is a multi-sampling, structured interview format in which interviewers, referred to as “raters,” assess specific applicant attribute(s) using multiple 5–15 minute interview stations. Each interview station is assigned a different discussion prompt, referred to as a “scenario;” likewise, each station has a different rater who is tasked with assigning applicants scores for a set of items on an evaluation tool (Eva et al. 2004c; Pau et al. 2013). The MMI’s multi-sampling technique has been celebrated for increasing the low reliability estimates that plague traditional MSPIs (Eva et al. 2004b, c; Uijtdehaage et al. 2011), and some studies suggest that MMI scores can be used to predict performance during medical school clerkships and on medical licensure examinations (Eva et al. 2004a, 2009, 2012).

MMI의 신뢰도는 흔히 G theory를 이용해서 추정된다. 어떤 측정이든 (MMI를 포함하여) 그 목적은 진점수(true score)를 흐릿하게 만드는 원하지 않는 변이(unwanted variance)를 줄이는 것이다. G theory를 활용한 대부분의 MMI에 대한 연구에서 그 모델은 평가자와 스테이션을 facet으로 모델링하였다.

Reliability of the MMI is commonly estimated using Generalizability (G) theory because of the multi-faceted nature of the measurements. In any measurement process, including the MMI, the goal is to reduce the unwanted variance in observed scores that can obscure true scores. G theory can simultaneously capture multiple sources of unwanted variance, referred to as “facets,” to provide an estimate of generalizability – or reliability (Brennan 2001). Consequently, MMI reliability is expressed as a G coefficient and represents a “universe of admissible observations” – a “universe” that is defined by the specific facet(s) that the researcher decides to include in the model. The decision regarding facets for inclusion is based on the context of a measurement to which the researcher plans to generalize findings. For instance, if some raters are always more lenient or more severe than other raters, then raters are a source of unwanted variance in MMI scores, and the rater facet would be modeled in a subsequent G study if the researcher wishes to generalize across raters. Most MMI studies associated with medical school admissions have modeled raters and/or stations as facets (Eva et al. 2004b, c; Uijtdehaage et al. 2011).

각 facet은 condition으로 구성되는데, 조건(condition)은 CTT에서 factor의 수준에 해당하는 것이다. 연구자들은 Condition에 대해서, condition은 어떤 측정의 질을 낮추지 않으면서도 바꿀 수 있다라고 가정한다. 

Each facet that defines the universe of admissible observations is comprised of “conditions” (Brennan 2001). These conditions are analogous to the levels of a factor in classical test theory (CTT). Overall, the MMI literature reports facets with a range of corresponding conditions and it is presumed, as an assumption for most applications of G theory, that the varying conditions represent random samples from these facets. Therefore, it is also generally assumed by researchers that these conditions can be altered without making the measurement any less acceptable. Although G theory can examine the extent to which such changes in a facet’s conditions make the measurement more or less acceptable (Shavelson & Webb 1991), this concept of interchangeability has not been examined for all potential facets of the MMI.

MMI에 대한 G coefficient연구에서 rater와 station을 facet으로 했지만, 어떤 인적특성을 평가하는가는 대체로 무시해왔던 것이 사실이다. MMI에서 평가하는 인적특성은 지금까지는 잘 포함시키지 않아왔지만 상당히 큰 변이(variance)의 원인이 될 수 잇다. 여러 연구에서 의사에게 요구되는 서로 다른 인적특성을 최대 87개까지 추출한 바 있지만, 입학면접에서는 그 중 일부만을 평가할 수 있을 뿐이다. 따라서 MMI에서 평가되는 구체적인 인적특성은 의과대학마다 리더십, 문화적 감수성, 대인관계, 비판적 사고 등으로 다양할 것이다. 또한 이들 평가는 주로 Likert scale로 평가하게 되며, 평가대상이 되는 인적특성은 의사로서 중요한 다양한 특성 중 무작위로 선정된다는 암묵적 가정을 기반으로 한다. 즉, MMI 연우게서는 이러한 인적특성들이 item 측면에서 상호교환가능함을 가정하고 있다. 즉, '리더십'에 대한 점수는 '문화적 감수성'에 대한 점수와 동등하고 상호교환가능하다는 뜻이다. 연구자들은 이러한 가정의 안면타당도에 의문을 제기하였으며 추가적 연구가 필요하게 되었다.

The extant literature reports moderate to high G coefficients for medical school MMIs ranging 0.58–0.81 (Eva et al. 2004b, c; Uijtdehaage et al. 2011). These reports, however, are based on studies that have modeled raters and stations as facets but have essentially ignored the impact of the attributes assessed. The attributes assessed by the MMI have the potential to introduce additional and largely unaccounted for, variance in MMI scores. The medical literature identifies up to 87 different attributes considered important for an aspiring physician (Price et al. 1971; Albanese et al. 2003); yet, an MSPI, including the MMI, can only reasonably capture a handful of these attributes. Therefore, the specific attributes assessed by an MMI will vary across medical schools and can range from leadership potential, cultural sensitivity, interpersonal skills, and critical thinking to a single, overall performance score (Eva et al. 2004c; Reiter et al. 2007; Uijtdehaage et al. 2011). These attributes are generally assessed as items on a Likert-like scale (Eva et al. 2004c) and carry the implicit assumption that an institution’s choice of attribute(s) can be considered a random selection from the domain of characteristics deemed important for the medical profession. Subsequently, MMI studies have largely considered attributes to be interchangeable conditions within the item facet. This would suggest that it is reasonable to believe that scores for the item, “leadership potential,” are parallel and interchangeable with scores for the item, “cultural sensitivity.” Consequently, the researchers question the face validity of this assumption and believe it warrants further investigation.

더 나아가서 MMI에서 평가하는 인적특성의 구성은 item facet을 넘어서 station facet으로 들어간다. 각 MMI 스테이션은 특정한 주제에 맞는 특정한 시나리오를 가지고 진행되는데, 이 시나리오를 바탕으로 평가서식에 의해서 'item'화 되는 사전에 결정된 인적특성에 대해 평가하게 된다. 결과적으로 station scenario 사이의 차이는 한번 더 의도하지 않은 변이를 유발할 수 있다. 기존 연구들은 station facet을 포함시키기는 했으나, station facet을 1회의 측정사건(measurement occasion)에만 국한시켰다. 따라서 기존 문헌에서 측정사건의 숫자가 증가할수록 MMI의 일반화가능도가 높아지는 것으로 되어있으므로 기존 연구의 결과는 CTT의 Spearman-Brown prophecy formula에만 부합하는 것일 수 있다. 

Furthermore, the impact of the composition of attributes assessed by the MMI has the potential to reach beyond the item facet into the station facet. Each MMI station is assigned a specific scenario that focuses on topics such as “knowledge of the healthcare system” or “critical thinking” and is intended to elicit information regarding a set of predetermined attributes that are captured as items on an evaluation form (Eva et al. 2004c). Consequently, differences among station scenarios have the potential to introduce further unwanted variance in the attributes assessed. Previous studies model the station facet into G studies (Eva et al. 2004b, c; Uijtdehaage et al. 2011); however, these studies generally recognize the station facet in terms of a measurement occasion only. Therefore, while it is well-established in the literature that increasing the number of measurement occasions increases generalizability estimates for the MMI, this finding merely aligns with the CTT’s Spearman-Brown prophecy formula. 

결과적으로 기존의 문헌은 스테이션의 시나리오가 MMI에서 인적특성을 평가하는데 미치는 영향력을 상당부분 무시해왔다고 볼 수 있다. MMI가 맥락특이성을 희석시키기 위한 목적으로 개발되었다는 점을 고려하면, 이에 대한 추가적 연구가 필요하다. 실제로 Eva의 파일럿연구를 보면, 지원자-스테이션 상호작용이 지원자 단독으로 인한 변이보다 다섯배나 컸다. 이러한 결과는 스테이션의 내용이 MMI점수에서 발생하는 오차의 중요한 원인이 될 수 있음을 보여준다. 그러나 아직까지 어떤 연구도 특정 인적특성(즉 item)에 대해서 MMI 스테이션의 내용, 즉 시나리오가 의과대학 지원자의 평가에 어떤 영향을 주는가를 연구한 바는 없다. 본 연구에서는 MMI평가서식의 구체적 특성에 의해서 정의내려진 item이 시나리오에 관계없이 여러 MMII station에 걸쳐서 일관되게 평가되어지는지를 연구해보고자 한다.

Consequently, the extant literature has largely ignored the potential influence of the stations’ scenario on the assessment of attributes within an MMI. Given the fact that the MMI was created in large part to dilute the effects of context specificity (Eva 2003), this warrants further investigation. In fact, Eva et al.’s (2004c) pilot study concluded that variance attributable to the candidate–station interaction was five times greater than that assigned to the candidate alone. This finding suggests that station content may introduce the most significant source of error in MMI scores. Yet, to the best of the authors’ knowledge, no study has examined how the MMI station content – the scenario – may influence the assessment and evaluation of medical school applicants on a set predetermined attributes (i.e. items). This study will explore whether items, defined as specific attributes on an MMI evaluation form, are assessed consistently across MMI stations regardless of station scenario.

Methods

This study examines one aspect of psychometric evidence from one United States (US) medical school that has fully adopted the MMI process as a replacement for the traditional MSPI. Using G theory, this study examines the variance attributable to the item facet and the scenario-item interaction. Data used for analysis represent MMI scores that were collected for the sole purpose of making admissions decisions. These data come from a US medical school that receives approximately 4000 admissions applications annually and interviews approximately 625 applicants each year. This institution fully adopted the MMI to select the entering class of 2009. With IRB approval (# 10-06-08-01), all applicants who participated in the MMI from 2009 to 2013 are included in the dataset used for analysis. This empirically collected dataset represents a nested design; therefore, only a small subset of applicants was used in this analysis in order to create a fully crossed design because in G theory, nested facets make it impossible to estimate all variance components separately (Brennan 2001).

'의사소통'과 이를 평가하기 위한 여섯 개의 구체적 특성 + 하나의 총괄평가

After a comprehensive blueprinting process, the school’s Admissions Committee identified one overarching characteristic – communication – to assess through the MMI. The rationale for choosing this single construct was largely rooted in literature that suggests that one of the chief patient complaints concerns poor communication between the patient and physician (Wofford et al. 2004). Communication was selected as the single construct from the larger domain of attributes deemed important for an aspiring physician. To operationalize this construct, six specific attributes and one “overall score” were used as sub score items in the MMI evaluation tool. The specific attributes assessed by this MMI included (1)multiple perspectives, (2)reflection of scenario, (3)articulation, (4)interest in dilemma, (5)non-verbal communication, and (6)interpersonal skills. These seven items were measured on a seven-point Likert-like scale that assumes equal intervals between the anchors (Unsatisfactory-1; Below Average-2; Slightly Below Average-3; Average-4; Slightly Above Average-5; Above Average-6; Outstanding-7).

G-String IV software (Bloch & Norman, Hamilton, Ontario, Canada), was used to estimate variance components attributable to the facets of measurement. In G theory, the object of measurement is not considered a facet. Therefore, this study’s two-facet design includes the object of measurement – applicants (p) – and two facets of generalization – scenario (s) and item (i).

Facet of differentiation

The object of measurement is considered the facet of differentiation. This facet of differentiation is analogous to the dependent variable and is considered the only desired source of variation. In other words, the object of measurement is the “universe” or “true” score (Brennan 2001). The facet of differentiation is the person, the applicant (p) facet, which represents the true MMI score for the applicant. Therefore, this variance should be large and other modeled sources of variance are expected to be small.

Facets of generalization

The facets of generalization are analogous to the independent variables and they contribute unwanted sources of error to the universe score, or for this study – MMI scores. These facets of generalization include the sources of measurement error that the researcher intends to generalize from the sample to the universe of admissible observations. Because this study intends to generalize applicant scores from one scenario to applicant scores from a much larger set of scenarios, scenario (s) is considered a facet of generalization. Likewise, because this study intends to generalize from applicant scores on one attribute item to applicant scores on a much larger set of items, item (i) is also a facet of generalization. In line with G theory assumptions, both the scenario facet and the item facet are considered random and conditions within these facets are deemed interchangeable (Shavelson & Webb 1991).

Confounded facet

Because there is one rater assigned to each scenario at this US medical school, the variance attributable to rater cannot be disentangled from variance attributable to scenario. Therefore, rater and scenario variance are completely confounded. For the purposes of this study; however, this confounded effect will be recognized as a limitation and the variance accounted for by this confounded facet will be considered attributable to the scenario (s).

Sample

지원자에 대한 MMI점수가 서로 다른 시나리오 아래서 수집되는 nested structure이다. 따라서 scenario facet이 applicant facet과 fully crossed 되는 subset을 찾는 purposive sampling을 하였음. 결과적으로 completely crossed design을 위하여 동일한 시나리오에서 동일한 아이템으로 평가받은 지원자를 표집하였다.

This study uses actual admissions data; therefore, the data structure represents a pragmatic design in which inevitable nesting and confounding occurs. A fully crossed G study elicits the most information; however, the existing data set represents a nested structure in which MMI scores are collected for applicants by using different scenarios. Therefore, a purposive sampling method was employed to generate a subset of data in which the scenario (s) facet (confounded but representing the same raters within scenario combinations) was fully crossed with the applicant (p) facet. Consequently, a subset of the full dataset was intentionally sampled for applicants rated within the same scenario using the same items to ensure a completely crossed design. The sample included 16 applicants who were evaluated within the same six scenarios and scored on the same seven items. This small, purposive sample was necessary in order to examine the variance attributable to the main effect of the scenario (s) facet and the scenario- item (si) interaction (Shavelson & Webb 1991), which is information pertinent to the study’s objectives.

Results

While the true score (p) should represent a sizable amount of variance, Table 1 shows that the applicant (p) represents only 6% of total variance. The estimated variance components from the G study suggest that the greatest amount of variance is attributable to the main effect of the scenario (s) facet and the interaction between scenario and applicant (ps). Collectively, these two variance components account for 77% of the total variance. The item facet (i) represents the lowest estimated variance, estimating only 0.6% of the total variance in MMI scores. Likewise, the scenario-item interaction (si) accounts for only 1.4% of the total variance. The low estimate of variance attributable to the item facet is reinforced by a high Cronbach’s alpha (0.97) for the seven items, which suggests very high internal consistency among the attributes measured by this MMI.

Discussion

일곱 개의 sub scores (items)의 높은 내적 일관성으로부터 현 MMI에서는 하나의 단일한 차원의 인적특성을 평가하고 있음을 알 수 있다. item facet으로부터 유발되는 variance가 2%에 불과한 것도 이를 지지한다. 만약 이 item들이 하나의 단일차원의 특성을 평가하는 것이라면 일곱 개의 item은 하나의 item으로 압축될 수 있다. p와 i에 의한 변이가 적다는 점은 대부분의 변이가 s에 기인한다는 것을 의미한다.

The high internal consistency of the seven sub scores (items) may support assumptions that the current MMI process is measuring one unidimensional attribute; this is further supported by only 2% of variance attributable to the item facet– (i), (pi), and (si). These findings either support the G theory assumption that conditions of the item facet can be considered interchangeable or it may suggest that raters do not understand how to use the items associated with the MMI evaluation tool and simply assign the same value for each item. If the items are capturing one unidimensional attribute, then a seven-item evaluation tool could be condensed into a single item tool. The low percentage of variance attributable to both items (i) and the true score – the applicant (p) – further suggests that the variation in MMI scores is mostly attributable to scenarios (s). 

스테이션 시나리오의 내용에 차이가 있다는 점을 감안하면, 시나리오에 의해서 지원자가 보여주는 인적특성의 비일관성이 높아진다고 보는 것이 타당하다. 예컨대 지원자는 윤리적 딜레마를 포함하고 있는 시나리오와 팀워크 활동을 포함하는 시나리오에서 서로 다른 특질을 보여줄 것이다. 따라서 MMI가 item차원에서는 하나의 단일차원 특성(one unidimensional attribute)을 측정하게끔 한다 하더라도, 스테이션의 내용은 그 특성(attribute)에 대한 측정을 변화시킴으로서 시나리오 수준에서의 다차원(multidimensionality)을 유발 할 수 있다. 연구자들은 이러한 차이가 시나리오-아이템 상호작용으로부터 나타날 것으로 기대했으나 본 연구의 결과는 이러한 가설이 틀렸음을 보여준다. 아마도 item facet으로 인한 변이의 비율이 낮기 때문에 이런 결과가 나왔을 것이다. 따라서 시나리오-아이템 상호작용이 작다는 것은 item facet으로 인한 variance가 scenario facet에 의한 variance에 포함되어버리기 때문일 수 있다. 결과적으로 이러한 상호작용이 MMI점수의 variance중 상당한 부분을 차지하게 될 것이나, 이러한 것이 이번 연구 샘플에서는 드러나지 않았다. 

Given the variation among the content of station scenarios, it is plausible to believe that scenarios promote inconsistencies among attributes exhibited by an applicant. For instance, a scenario involving an ethical dilemma might highlight different attributes than a scenario requiring an applicant to engage in a teamwork activity. Therefore, even if the MMI is supposedly measuring one unidimensional attribute at the item level, the content of the stations may elicit different measurements of attributes, thereby introducing multidimensionality at the scenario level. While the researchers expected to find this disparity manifested as a large variance component associated with the scenario-item interaction, this initial analysis does not support the assumption. This potential effect may be obscured by the low percentage of variance attributable to the item facet. Therefore, it is possible that the small scenario-item interaction is a result of variance attributable to the item facet being subsumed by the variance attributable to the varying conditions of the scenario facet. Consequently, the interaction may indeed contribute substantial variance in MMI scores; but this was not identified within this study’s sample.

AERA, APA, NCME 기준에서 드러나듯, "만약 문항 개발자가 시험을 수행하는 조건이 응시자에 따라서 다를 수 있음을 적시한다면, 그러한 조건에서 허용가능한 변이가 확인되어야 하고, 서로 다른 조건을 인정하는 rationale가 명시되어야 한다"라고 언급하고 잇다. item facet으로부터 기인하는 변이가 작다는 것이 item facet이라는 서로 다른 조건에 대한(즉 서로 다른 특성들에 대한) 허용가능성을 의미할 수 있지만, 이러한 가정은 scenario facet에는 해당되지 않는다. 본 연구의 결과는 scenario facet이라는 조건에 대한 상호교환가능성에 의문을 제기한다. 따라서 시나리오 선정에 보다 주의를 기울일 필요가 있다.

As outlined by the AERA, APA and NCME Standard 3.21, “If the test developer indicates that the conditions of administration are permitted to vary from one test taker or group to another, permissible variation in conditions for administration should be identified, and a rationale for permitting the different conditions should be documented” (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education 1999, p. 47). While the low variance attributable to the item facet may suggest permissibility for permitting different conditions of the item facet (i.e. attributes), this assumption may not hold for the conditions of the scenario facet. The results of this study suggest that interchangeability for the conditions of the scenario facet is questionable. Subsequently, more attention should be directed towards the selection of scenarios.

이러한 연구결과에도 불구하여 몇 가지 한계가 있다.

Despite these findings, this study has some limitations. Because this study uses empirically collected data intended for admissions purposes, the researchers did not have direct control over the data collection process. Consequently, this G study is limited by the need for a purposive sample which results in a very small sample size relative to the larger dataset. The researchers felt that the benefit of using a fully crossed design justified the small sample size. In G theory, the nested facet cannot be estimated separately from its interaction effects because nesting creates missing cells in the design; additionally, the scenario-item (si) interaction, a major focus of this study, could only be examined using a fully crossed design (Brennan 2001). Nonetheless, the sample size used in this study may not be representative of the larger population; subsequently, the variance components might be influenced by sampling error. Because estimated variance components can be very unstable when the number of conditions within a measurement is small, this study could be replicated using a larger sample size. In addition, only two facets of generalization are modeled and one of these facets, the scenario facet, is confounded with another potential source of variation – the rater facet. Therefore, other facets could be added to the model in order to expand the universe of admissible observations and the corresponding generalizability of the study. Consequently, this study’s external validity is limited to the extent that other MMIs mirror the one used in this analysis. Despite these limitations, this study offers a solid framework for future exploration into the impact that scenario content can have on the attributes assessed by the MMI.

Conclusions

Because of the variation in how and what an institution-specific MMI measures, psychometric properties must be examined for each medical school that chooses to adopt the MMI as a replacement for the MSPI. This study adds to the growing body of literature related to psychometric analyses of the MMI. Because the extant literature has primarily focused on predictive validity and largely ignored other aspects of validity, this study adds to the foundation for further exploration into construct validity. As the MMI continues to gain momentum as a replacement for the traditional MSPI, the measurement process deserves careful attention, especially in terms of how and what is measured. Future analysis should explore the potential that both items and scenarios have on subsequent MMI scores. Overall, the results of this study reinforce the need to examine all psychometric properties of a measurement process – especially one, such as the MMI, that is used for high-stakes admissions purposes.

Variance in attributes assessed by the multiple mini-interview.

불명료함을 견디는 능력: 의과대학 학생선발의 윤리기반 준거 (Academic Medicine, 2013)

Tolerance for Ambiguity: An Ethics-Based Criterion for Medical Student Selection

Gail Geller, ScD, MHS

몇 해 전, 저자는 의료윤리 과목에 대한 학생들의 반응을 흥미롭게 지켜본 적이 있다. 학생들이 의료의 '불확실성'에 대응하는 방법에 확연한 차이가 있었다. 

Several years ago, I coordinated the ethics course that was required for first year medical students at my institution, the Johns Hopkins School of Medicine. I was keenly aware of significant differences in students’ reactions to the course. (...) Intrigued by what I noted as variability in students’ tolerance for ambiguity, I searched for and discovered a substantial social science literature on this topic. (...) The way students respond to uncertainty in medicine deserves heightened attention in light of imminent changes to the medical student selection process, which motivated me to write this Perspective

임박한 의과대학 입학절차 변화 Impending Changes to the Medical School Admission Process

AAMC는 의과대학 입학절차를 바꾸고자 지난 10년간 많은 노력을 해왔으며, 그 목표 중 하나는 인문학적 특성을 평가하여 다른 사람과 의사소통을 잘 하고, 환자와 좋은 관계를 맺을 수 있으며, 윤리적 판단을 내릴 수 있는 잠재력을 가진 학생을 선발하자는 것이다. MCAT시험에 사회과학과 행동과학 내용을 추가하도록 결정되었고, 의과대학 지원시 선수과목으로 요구하도록 했다. 2015년부터 도입 예정이다.

For the last decade, the Association of American Medical Colleges has been interested in and committed to transforming the medical school admission process. The goal is to enable the assessment of humanistic characteristics and, thus, to select students who are more likely to become physicians who can communicate and relate with patients and engage in ethical decision making. Recently, the decision was made to revise the MCAT exam to include more social and behavioral science and to adjust the prerequisite course requirements for admission to medical school.2 These changes will be implemented in 2015.

The Concept of Tolerance for Ambiguity

지난 몇 년간 의학교육과 의료에 있어서 '불명료함'과 '불확실성'의 영향에 대한 많은 연구가 이뤄져왔다. 이 두 용어가 혼재되어 사용되긴 하지만 동일한 개념은 아니다. Ellsberg는 두 가지가 모두 어떤 "risk"의 한 종류지만, 그 "가능성"에 있어 차이가 있다고 하엿다. 즉, '불확실성(uncertainty)'는 어떤 결과가 일어날 가능성을 아는 것이며, '불명료함(ambiguity)'은 어떤 결과가 일어날 가능성을 모르는 것이라고 구분하였다. Grenier는 시간에 따른 구분을 제시했는데, 불확실성은 미래에 일어날 일에 대한 것이며, 불명료함은 현재의 상황에 대한 것이다. 이러한 것들을 고려한다면 '불명료한' 상황이 조금 더 모호한 상황을 만들며, 더 시급한 것이고 따라서 더 많은 tolerance를 요구한다.

In the past several years, there has been extensive scholarship on the impact of ambiguity and uncertainty on medical education and medical care. Although  these concepts are related and have beenused interchangeably, ambiguity and uncertainty are not equivalent.3,4 Ellsberg5 writes that both are types of “risk,” but they vary in probability: In a case of uncertainty, the probability of a particular outcome is known; with ambiguity, the probability is unknown. Grenier et al3 propose a time-oriented distinction, with uncertainty relating to an event in the future and ambiguity concerning circumstances in the present. In this light, “ambiguous” situations have either more shades of gray or greater urgency and may, thus, require more tolerance.

여러 문헌에서 불명료함을 회피하고자 하는 성향이 가져올 수 있는 부정적 결과가 연구된 바 있다.

It is also important to note the recent literature on ambiguity aversion and its adverse consequences in both medical practice6 and clinical research.7

불명료함을 견디지 못하는 것, 혹은 회피하려고 하는 것에 대해서 처음 언급된 것은 50년도 더 전이다. 처음에 이러한 특성은 '새롭거나 복잡하고 정답이 없는' 상황을 '위협의 근원'으로 받아들이는 것으로 묘사되었다. 보건의료는 그 특성상 새롭고, 복잡하고, 종종 정답이 없기 때문에, 어떻게 그러한 상황에서 의사들이 반응하는지를 이해하는 것이 중요하다. 불명료함에 대한 내성은 권위주의, 독단주의, 완고함, 규정에 대한 순응, 윤리적 편견 등과 관련이 있다. 이러한 특징은 명백히 휴머니즘, 문화적 역량, 환자중심 등과 대치되는 개념이다.

Intolerance of ambiguity, or aversion to ambiguity, was first identified more than 50 years ago.8 It was described as a personality characteristic in which situations that are “novel, complex or insoluble” are perceived as “sources of threat.”9 To the degree that medicine and health care are characterized by novelty, complexity, and sometimes insolubility, it is extremely important to understand how clinicians react to such circumstances. 

• In general, individuals with high ambiguity tolerance are drawn to or captivated by the unknown. 
• By contrast, those with low tolerance tend to deny, avoid, or minimize ambiguity, and experience significant stress when faced with it.9 

Ambiguity intolerance has been associated with other personality traits such as authoritarianism, dogmatism, rigidity, conformity, and ethnic prejudice.9,10 Clearly, these traits contradict the humanistic, culturally competent, and patient-centered qualities underlying ethical medical practice.

Tolerance for Ambiguity in Medical Practice and Education

불명료함에 대한 내성은 의대생들의 태도와 행동에 큰 영향을 준다. 의과대학생의 불명료함에 대한 내성에 관한 연구에 따르면 여러 사회인구학적, 행동적 특성과 연관되어 있다. 이러한 근거를 따르면 학생의 높은 불명료함 내성은 리더십 능력, 농촌지역에서의 근무의사 등과 상관이 있다. 반대로, 낮은 내성은 실수에 대한 두려움, 취약계층에 대한 부정적 태도, 알콜 남용자에 대한 편견과 높은 상관이 있다. 불명료함에 대한 내성이 전공 선택과 관련이 있는지 여부는 불확실한데 일부 연구에서 관련성이 없다고 보고된 바 있고, 다른 연구에서는 관련성이 보고된 적도 있다.

Tolerance for ambiguity also exerts a powerful influence on the attitudes and behaviors of medical students. Numerous studies have measured students’ levels of ambiguity tolerance1,19–21 and correlated their scores with a range of sociodemographic and behavioral characteristics.19–26 This evidence suggests that higher tolerance for ambiguity is associated with students’ leadership ability25 and their willingness to practice in rural areas.26 Conversely, there is a strong relationship between students’ low tolerance for ambiguity and their fears of making mistakes,22 their negative attitudes toward the underserved,23,24 and bias against those who abuse alcohol.1 It remains unclear whether tolerance of ambiguity is linked to students’ specialty choices. In some studies, there was no association.20,21 In others, specialties that require high levels of precision, such as surgery, tended to attract individuals with low ambiguity tolerance. Conversely, specialties that are inherently ambiguous, such as psychiatry, appealed to individuals with higher tolerance.1

이러한 중요도에도 불구하고 불명료함에 대한 내성은 의과대학생 선발이나 교육과정에서 모두 간과되어왔다. 사회학자들은 오래 전부터 의학에 대해서 '확실한 것'을 보상하는 경향이 있음을 지적한 바 있다.불명료함과 불확실성이 의과대학의 문화에서 배제되고 있음을 인정한다면, 불명료함에 대한 내성을 가르쳐야 한다는 최근의 제안도 받아들여질 필요가 있다. 레지던트를 대상으로 한 연구를 살펴보면, 긴 시간에 걸쳐서 불명료함에 대한 내성을 기를 수 있는 것으로 보이지만, 이 문제는 의과대학생 수준에서 연구된 바 없다. 

Despite its importance, tolerance for ambiguity has been overlooked both in the selection and also the training of medical students.27–30 Sociologists of medicine have long observed that the medical education process rewards certainty.27,28 In recognition that ambiguity and uncertainty have been neglected in the culture of medicine, there have been recent proposals to acknowledge, embrace, and explicitly cultivate ambiguity tolerance in the medical curricula.29,30 This is undoubtedly a laudable goal, but it assumes that ambiguity tolerance can be taught. Although evidence among residents suggests that ambiguity tolerance can improve over time and with experience,31 this question has not been explored among medical students. Studies of ambiguity tolerance in medical education have been cross-sectional, not prospective.

하나의 검증해볼만한 가설은 불명료함에 대한 내성이 인적 특성인지, 일시적인 상태인지를 보는 것이다.

One testable hypothesis is that tolerance for ambiguity is both a personality trait and a temporal state.

문헌에 근거해서 판단해보자면, 의과대학 입학시에 불명료함에 대한 내성이 높은 학생은 의학과 의료의 불확실한 특성에 점차 자극을 받는 반면, 내성이 낮았던 학생은 이런 상황을 더 회피하려고 하게 된다는 가설이 합당하다.

Based on the conceptual literature, a reasonable hypothesis is that students who enter medical school with high tolerance for ambiguity are drawn to, and stimulated by, the uncertainties that characterize medicine and patient care. (...) By contrast, students who enter medical school with low tolerance for ambiguity may be more likely to avoid, minimize, or negate the uncertainties that characterize medicine and patient care.

A Timely Proposal

질적, 양적, 혼합적 전략이 모두 필요하다. 타당도가 검증된 검사들이 존재한다.

The assessment plan could consist of quantitative strategies, qualitative strategies, or a combination of both. With respect to quantitative strategies, a number of validated scales exist1,3,6,9,10,32,33 that could be used or adapted for use in the medical admission process.

학업적 기준을 만족하는 학생들 중 불명료성 내성이 일정 수준 이상인 학생만 면접할 수도 있고, 모두 면접대상자로 선발한 다음에 불명료성의 내성을 검사할 수도 있다.

Among students who otherwise meet the academic criteria for admission, one option would be to offer interviews only to those whose tolerance scores exceed a certain cutoff. An alternative strategy would be to offer interviews to all students who meet the academic standards for admission and, during the interview, use the tolerance scores to explore, qualitatively, students’ own assessments of their tolerance for ambiguity.

Team care, IPE와 같은 의학교육 분야의 엄청난 문화적 변화가 진행중이고, 이러한 변화를 따라가기 위해서는 학생들도 불명료함에 대한 내성을 더 기를 필요가 있다. 왜냐하면 팀의 구성원이 모든 결정에 다 동의하는 것이 아니기 때문이다. 현재까지 학생들에게 동료의 의견을 존중해야 함을 가르치고 있지만, 학생들이 그룹 단위의 의사결정에 내재된 '불명료함'에 대해서 얼마나 내성이 있고 편안하게 느낄 수 있는가는 모를 일이다.

There are already significant culture changes under way in medical education, such as the growing emphasis on team care and interprofessional education. These changes may require greater tolerance for ambiguity among students because, occasionally, members of the team will disagree. Although students are being taught to “respect” their colleagues’ opinions (i.e., listen openly and not criticize), they may not be comfortable tolerating the ambiguity inherent in group decision making.

Tolerance for ambiguity: an ethics-based criterion for medical student selection.

도덕지향에 따른 의과대학생 선발(Medical Education, 2005)

Selection of medical students according to their moral orientation

Miles Bore,1,2 Don Munro,2 Ian Kerridge3 & David Powis1

INTRODUCTION:

Consideration has been given to the use of tests of moral reasoning in the selection procedure for medical students. We argue thatmoral orientation, rather than moral reasoning, might be more efficacious in minimising the likelihood of inappropriate ethical behaviour in medicine. A conceptualisation and measure of moral orientation are presented, together with findings from 11 samples of medical school applicants and students.

AIM:

To provide empirical evidence for the reliability and validity of a measure of moral orientation and to explore gender, age, cultural and educational influences on moral orientation.

METHODS:

A questionnaire designed to measure a libertarian-dual-communitarian dimension of moral orientation was completed by 7864 medicalschool applicants and students in Australia, Israel, Fiji, New Zealand, Scotland and England and by 84 Australian psychology students between 1997 and 2001.

RESULTS:

Older respondents produced marginally higher (more communitarian) moral orientation scores, as did women compared to men. Minor but significant (P <0.05) cultural differences were found. The Israeli samples produced higher mean moral orientation scores, while the Australian psychology student sample produced a lower (more libertarian) mean score relative to all other samples. No significant change in moral orientationscore was observed after 1 year in a sample of Australian medical school students (n=59), although some differences observed between 5 cohorts of Australian medical students (Years 1-5; n=234) did reach significance. Moral orientation scores were found to be significantly correlated with a number of personality measures, providing evidence of construct validity. In all samples moral orientation significantly predicted the moral decisions made in response to the hypothetical dilemmas embedded in the measurement instrument. Discussion The results provide support for the conceptualisation of a libertarian-dual-communitarian dimension of moral orientation and demonstrate the psychometric properties of the measurement instrument. A number of questions concerning the use of such tests in selection procedures are considered.

Introduction

최근 CMA Journal에 의과대학생이 학년이 올라감에 따라 도덕적 추론의 단계가 하강하는 것 같다는 연구결과에 대하여 commentary를 하면서 Singer는 윤리적인 의사를 선발하고 교육하는 것의 복잡함을 지적한 바 있다. 특히 Singer는 의과대학생을 논리적추론능력에 기반하여 선발해야한다고 주장하였다. 이러한 주장이 일견 타당해보이지만, 우리는 도덕적추론에 대한 시험을 선발도구로 사용하는 것이 부적절하며, 그보다는 도덕지향(moral orientation)을 사용하는 것이 더 좋음을 주장하고자 한다.

In a recent commentary in the Canadian Medical Association journal, written in response to findings that students' stages of moral reasoning appeared to decline over the course of medical education,1 Singer2 addressed some of the complexities of selecting and training ethical practitioners. In particular, Singer suggested the need for the selection of medical students to be based on level of moral reasoning. While this seems reasonable, we would argue that tests of moral reasoning are inappropriate for use as selection instruments and that it may be more valuable to consider individual differences in ‘moral orientation’.

Moral reasoning

도덕 심리학과 도덕적 추론과 관련한 기존의 연구는 콜버그의 이론을 기반으로 하고 있다. 콜버그는 도덕성 발달의 인지적 발달이론을 주창한 사람으로서, 6단계의 도덕적 정의추론 단계를 구성하였다. 이러한 단계는 개개인이 추상적인 윤리적 이론을 실제 윤리적 문제에 적용하여 활용하는 인지적 발달단계이며, 콜버그의 도덕적 추론에 대한 이론은 MJI, SROM, DIT 등 다양한 검사의 근간을 이루고 있다.

Much of the literature relating to moral reasoning and moral psychology is based upon the work of Lawrence Kohlberg,3,4 who outlined a cognitive developmental theory of moral development constructed upon 6 stages of moral justice reasoning (Table 1). These stages reflect an individual's progressive cognitive developmental ability to utilise abstract ethical principles in addressing ethical issues. Kohlberg's theory of moral reasoning forms the basis of a number of tests of morality, including the Moral Judgement Interview (MJI),5 the Sociomoral Reflective Objective Measure (SROM)6 and the Defining Issues Test (DIT).7

콜버그의 이론과 척도가 널리 받아들여지고 있지만 여러가지 의문도 제기된 바 있어서, 문화적, 특히 서구문화의 bias가 작용할 수 있다는 점, 6단계와 일부 5단계에 대한 근거가 부족하다는 점, 성별에 따른 bias가 존재할 수 있다는 점, MJI의 점수산출방법, 도덕추론단계가 실제 행동을 예측하는 능력 등등이 지적되어왔다. 그러나 도덕추론에 대한 척도(MJI, DIT, SROM등)를 개발하는데 있어서 보다 근본적인 문제가 있다. 응답자는 가상의 딜레마 상황을 제시받게 되고, 도덕적 판단을 내린 뒤 응답자가 그렇게 응답한 근거를 선택하게 되어있다. 이 때 근거로 제시한 이유에 따라서 도덕적 추론능력 점수가 매겨지는데, 따라서 이러한 검사가 보여주는 것은 의사결정을 내린 후 사후에 그것을 정당화하는 능력이다. 콜버그의 이론이 한 개인의 도덕적 가치관이나 신념에 대해서 다뤘다기보다는 추상적인 윤리적 원칙을 활용하는 능력에 대한 발달단계를 다루고 있다는 점에서 위 척도들의 설계는 적절하며 논리적인 것으로 보인다. 한 개인이 교육을 받고 성숙해감에 따라서 추상적 이론을 실제 도덕적 의사결정에 활용하여 타당성을 입증할 수 있다.

While Kohlberg's theory and measure are highly regarded, questions have been raised about cultural or Western liberal bias,8–11 a lack of evidence for stage 6 and to some extent stage 5,12 a possible gender bias,13 the scoring method of the MJI,14,15 and the ability of moral reasoning stages to predict behaviour.14,16,17 There is, however, a further, more fundamental problem with the design of major measures of moral reasoning (e.g. the MJI, DIT and SROM). In each of these instruments, respondents are presented with a hypothetical dilemma and asked to make a moral decision concerning the dilemma and to give reasons for their decision. Moral reasoning stage scores are determined from the reasons they provide. Thus, these tests appear to provide a measure of the ability to produce post-decisional justifications. Given that Kohlberg was not concerned with a person's moral values and beliefs (cognitive content), but with the development of the capacity to utilise abstract ethical principles (cognitive structure), the measurement design is appropriate and logical. As individuals mature and gain greater levels of education,18 they become increasingly able to utilise abstract principles to justify their moral decisions.

그러나 자신의 결정이 도덕적으로 옳음을 보여주는 능력은 실제로 그 행동이 도덕적인가와는 다를 수 있다. 역사적으로 이것을 보여주는 무수한 사례가 있으며, 최근 이라크 침공에 대한 보도도 그것이 정당하다는 기사와 정당하지 않다는 기사가 모두 존재한다. Bandura는 '비인간적인 행동의 도덕적 정당성을 입증하는 것은 어렵지 않다'라고 했다.

The ability to justify a decision, however, may have little to do with whether that decision or subsequent behaviour was actually ‘moral’. History is replete with examples of individuals who committed atrocities but were quite able to justify their actions by reference to normative ethical theories or to ethical principles. More recently, the media has reported principled justifications for invading Iraq and principled justifications for not invading Iraq. As Bandura19 notes, ‘It is not uncommon for sophisticated moral justifications to subserve inhumane endeavours.’

도덕적 의사결정이 이미 내려진 의사결정의 정당성을 입증하는 것과 무관하다는 것은 비인간적인 행동에만 국한된 것은 아니다. 콜버그의 이론에서 최상위 단계의 근간인 칸트의 윤리원칙을 사용해서 인공호흡기를 유지할 것이라는 결정 뿐 아니라 유지하지 않을 것이라는 결정도 정당화할 수 있다.

The independence of a moral decision and the ability to justify the decision are not limited to inhumane endeavours. For example, the decisions to discontinue or to continue mechanical ventilation can both be justified using the abstract Kantian ethical principles that underpin Kohlberg's highest stage of moral reasoning ability.

의사결정을 내리고 거기에 대해서 타당한 근거를 제시하는 것에 초점을 두기보다는, 개개인이 의사결정을 내릴 때 어디에 근거해서 그러한 결정을 내리는가에 초점을 맞추는 관점이 필요하다. 말하자면 '어떤 심리학적 변인이 윤리적 민감성, 도덕적 의사결정에 대한 개인간 차이를 만들어내는가?' 우리는 세 가지를 주장하고자 한다.

Rather than focussing on the justifications that an individual might give for their decisions, an alternative view is to consider what it is about an individual that determines their opinions, their decisions and their actions. To put this in question form: what psychological variables lead to individual difference in ethical sensitivity (the recognition of an ethical situation), moral decision making, the decisions made, and the interpersonal behaviours displayed in making and enacting the moral decision? We would argue that 3 factors are highly relevant:

1. 개개인의 도덕적 지향

2. 도덕적 행동과 관련한 인적 특성

3. 도덕적 규준/정책/전문원칙에 대한 개인의 지식과 경험 

1. an individual's moral orientation;

2. personality traits that may influence moral decision making and the performance of moral behaviour, and

3. an individual's knowledge and experience of moral norms, laws, policies, professional principles and the professional culture in which the person is operating.

도덕지향 Moral orientation

도덕지향에 대한 개념은 Gilligan에 의해서 처음 제시되었는데, 그는 여성이 보다 care-oriented되어있고, 남성은 justice-oriented 되어있다고 주장하엿다. 연구를 바탕으로 한 명확한 근거가 존재하지는 않지만, 도덕지향에 관한 개개인의 차이가 도덕적 행동에 영향을 줄 수 있다는 주장은 눈여겨볼 만 하다.

The concept of moral orientation was proposed by Gilligan,13 who suggested that women are more care oriented while men are justice oriented. While this hypothesis has not been clearly supported by empirical studies,20,21 the contention that individual differences in moral orientation might be influential in moral behaviour is noteworthy.

우리는 도덕적 의사결정을 내리기 전에 가지고 있던 도덕지향을 측정하기 위해 만들어진 설문지를 활용한 연구에 근거하여,도덕지향에 관한 또 다른 개념을 개발하고자 했다. 본 연구는 개개인의 도덕지향의 차이가 정규분포를 이루며, 한 극단에는 libertarian이, 다른 극단에는 communitarian이 있음을 보여준다. 대부분의 응답자는 그 중간 어디에 위치하며, 대체적으로 개인적 요구와 사회의 요구를 균등하게 고려한 의사결정을 내린다.

We have developed an alternative conceptualisation of moral orientation based on insights arising from empirical studies using a questionnaire designed to measure moral orientation prior to the making of a moral decision.22 This research indicated that individual differences in moral orientation formed a normally distributed trait-like dimension with, 

• at one extreme, respondents consistently placing greater importance on the needs, rights and well-being of individuals and relatively less importance on the rights, needs, norms and well-being of society and referent groups within society. We labelled this a ‘libertarian’ moral orientation. 
• The opposite was apparent at the other extreme of the dimension, with respondents consistently placing greater importance on group/society needs and relatively less importance to the needs of individuals: a ‘communitarian’ orientation. 
• A majority of respondents, occupying the central area of the score distribution, appeared to give approximately equal importance to individual needs and group/societal needs, indicating a ‘dual’ moral orientation.

도덕지향에 대한 이러한 관점에 따르면, 도덕적 딜레마 상황에서, 도덕지향은 맥락을 이해하고 처리하는데 관여하며 가능한 선택사항과 개인과 집단에 미치는 영향을 평가하여 의사결정을 내리게 된다. 

The articulation of this view of moral orientation is that, when presented with a moral dilemma, the moral orientation of the respondent mediates the perception and processing of the context, the evaluation of potential options and consequences for individuals and groups and determines/predicts the moral decisions the respondent makes. In short, when confronted by a moral dilemma, 

• libertarians will ‘see’ and place greater value on the needs of and potential consequences for the individual/s in the context, 
• communitarians will ‘see’ and place greater value on the needs of and potential consequences for society and important referent groups within that society, and 
• the dual-oriented will ‘see’ and approximately equally value the needs of and consequences for both the individual/s and society.

The above conceptualisation emerged in concert with the development of a questionnaire-based measure of libertarian−communitarian moral orientations that we called the Mojac Scale.

Mojac Scale의 점수가 관련되어있는 가치들을 살펴보면 construct validity의 근거도 어느 정도 있는 것으로 보인다. 또한 이 점수는 도덕추론 단계와는 상관이 없는 것으로 나타난다.

Scores from this scale have been found to be empirically related to the values of hedonism and social power (favoured by libertarians), beneficence and tradition (favoured by communitarians), thus providing some evidence of construct validity. Furthermore, the scores were found to be unrelated to moral reasoning stage.22,23

The study reported below aimed to examine the influence of education, age, gender and culture and the relationship of the libertarian−communitarian dimension to particular personality traits and the prediction of moral decisions.

Methods

Participants

연구참여자 

From 1997 to 2001 data were collected from 11 samples of applicants to medical schools and medical school students in Australia, England, Scotland, New Zealand, Fiji and Israel. The samples were chosen for the purpose of determining test norms and examining the variables of education, age, gender and culture in conjunction with a broader research project reported elsewhere.24 Sample description, size, age and gender details are shown in Table 2.

Instruments

두 가지 버전의 Mojac Scale 사용. 24문항의 짧은 버전과 45문항의 긴 버전.

All participants completed either the short (24 items) or long (45 items) measure of the Mojac Scale.22,23 The short measure (Mojac-24) consists of 3 hypothetical dilemmas (vignettes); Mojac-45 contains an additional dilemma. Respondents read each dilemma and then respond to a series of statements relevant to the needs of individuals or to the needs and moral expectations/norms of society using a 4-point Likert scale (strongly agree to strongly disagree). Respondents were also asked to make a forced choice 2-option ‘final decision’ for each dilemma. Responses to the 24 (or 45) statement items were used to derive a libertarian (low score) to communitarian (high score) moral orientation score (LibCom score). An example of the Mojac protocol, using a dilemma based on the ‘Heinz’ dilemma used in the MJI, the DIT and SROM is given in the Appendix.

H그룹과 I그룹은 다른 인성검사도 시행했음. J그룹도 다른 검사 시행. 

Samples H and I also completed the following personality tests: Right-wing Authoritarianism,25 Social Desirability,26 the International Personality Item Pool (IPIP) measure of the Big 5 Factors of Personality27 (extroversion, neuroticism, openness, agreeableness and conscientiousness), the NACE Scale24 (a measure of narcissism, aloofness, confidence and empathy), and the 16 Personality Factors28 (16PF) scale. Sample J completed the Sensitivity to Punishment and Sensitivity to Reward Scale29 (SPSRS), the IPIP and the Eysenck Personality Questionnaire30 (EPQ: Extroversion, Neuroticism and Psychoticism). These tests were chosen as the traits they measure (or specific traits within multi-trait scales) were expected to provide further evidence of the construct validity of the Mojac Scale. In samples A and G, participants also completed a battery of tests for the purpose of selection to medical schools; however, scores from the selection tests were not included in this study.

Procedure

절차

For all samples, the tests were administered under supervision in pen and paper format, using either optical mark reading (OMR) response forms or hand-marked forms, to participants in either a large hall or room. The response sheets were then collected and the data either scanned or hand-entered into spreadsheets for statistical analysis.

Results

Reliability

신뢰도

Cronbach's α reliability coefficients of 0.82−0.87 for the 24-item short form and 0.83−0.92 for the 45-item version were found (Table 2), indicating a high and stable internal consistency for the measure.

Age and gender

연령과 성별 - 약하지만 유의한 상관관계

Although the age distribution was greatly skewed in all samples, weak but significant (P < 0.05) positive correlations were found between age and combined Mojac-24 samples LibCom scores (r = 0.19) and combined Mojac-45 samples (r = 0.18). Weak but significant gender differences were also found. In the Mojac-24 samples the mean LibCom score for women (66.4, SD = 9.7) was significantly higher than the mean LibCom score for men (65.0, SD = 9.1; t = − 4.88, P < 0.001). This difference was also observed in the Mojac-45 samples (women 114.9, SD = 14.3; men 110.7, SD = 15.4; t = 8.51, P < 0.001).

Differences between samples

표집간 차이 - 거의 비슷했으나 일부 차이 있는 집단 존재

Generally, differences in the mean LibCom scores, standard deviation and range across samples (Table 2) were not large and a similar near-normal distribution was evident in all samples. Some differences did reach statistical significance as indicated by a 1-way analysis of variance (anova) and Tukey's post hoc pairwise comparisons with a family error rate of P = 0.05. For the Mojac-24 samples, the means from both Israeli samples were significantly higher than all other Mojac-24 samples, while the mean for the psychology students was significantly lower than all other Mojac-24 samples [F(5, 4227) = 30.1, P < 0.001]. A 1-way anova of the Mojac-45 samples also reached statistical significance [F(4, 3714) = 3.54, P = 0.007]; however, no significant difference between any pair of means was found.

Influence of medical education

교육의 효과: C그룹에서 1년 차이를 보았을 때는 거의 차이 없음. 

A subsample of sample C completed the Mojac-24 again 12 months after the initial testing in 1999. If medical education does influence moral orientation, then a significant difference in the 1999 and 2000 sample mean scores would be expected. For this subsample of 59 students, the 2000 LibCom mean of 64.0 (SD = 7.6) was not significantly different from the 1999 mean of 62.9 (SD = 7.3). The correlation between scores produced in 1999 and 2000 was r = 0.77, indicating only minor changes in moral orientation after 1 year. While this finding also suggests acceptable test-retest reliability, a study with a more typical period of 3−4 weeks between test and retest has not yet been undertaken.

C그룹에서 1,2,3,4,5학년을 비교했을 때 3학년과 5학년이 1학년보다 높음.

Sample C was of sufficient size to allow cross-sectional comparison of LibCom means between students from Years 1, 2, 3, 4 and 5 of the medicine programme, with 65, 43, 59, 30 and 37 students in each year cohort, respectively. anova indicated that there were significant differences between the year groups [F(4, 229) = 4.72, P = 0.001]. A Tukey's pairwise comparison of the means with a family error rate of P = 0.05 indicated that Year 1 participants had significantly lower scores than Year 3 and Year 5 participants. This is also indicated in the plot of the means and 95% confidence intervals shown in Fig. 1. No other significant differences between year levels were found.

학년은 LibCom score의 유의한 예측인자이나, 나이는 그렇지 않음.

A tendency for later-year students to produce higher LibCom scores (more communitarian) is apparent in Fig. 1. A regression analysis found that year of study was a significant predictor of LibCom scores (t = 2.85, P = 0.005), while age was not (P > 0.05). However, the variance of LibCom scores accounted for by the predictors was minimal (R-Sq = 4.2%; t = 3.18, P = 0.002). These findings indicate modest differences between the year cohorts tested. Observation of any change in moral orientation requires a longitudinal design and such a study is yet to be completed.

Construct validity

세 표집에서 construct validity를 확인해보았음. marker test와 비교했을 때 유의한 상관관계를 보임

The conceptualisation of Mojac scores as indicative of a continuum of libertarian to communitarian moral orientation was tested against several well validated personality measures in 3 samples: 508 Scottish medical school applicants (sample J) and 2 samples of New Zealand medical school students (samples H and I in which a total of 204 participants completed the same test battery). Table 3 shows highly significant (P < 0.001) correlations between Mojac-45 LibCom scores and scores from the ‘marker’ tests.

Predictive validity

예측타당도: Mojac Scale의 마지막 문항인 최종결정에 대한 점수와 LibCom score 비교하였을 때, 30%~40%의 변동을 설명할 수 있다.

The relationship between the libertarian−communitarian dimension and the moral decisions individuals make was examined using the final decision items embedded in the Mojac Scale. Final decision scores for each respondent were determined by coding the response options for each of the 3 (Mojac-24) or 4 (Mojac-45) final decision items as 1 for a decision that favoured the individual in the dilemma and 2 for a decision that favoured the group. The final decision items were then summed to produce an overall final decision score.

Regression analysis was used to examine the relationship between LibCom scores and final decision scores. Across all samples, LibCom scores were found to account for approximately 30% (Mojac-24) to 40% (Mojac-45) of the variance in final decision scores. R-Sq values for all samples are given in Table 4. Additionally, in the sample of 2906 Australian medical school applicants, 3 moral orientation groups were created using a tri-median split of LibCom scores. Figure 2 indicates that libertarian-oriented respondents showed a strong tendency to make decisions that favoured the outcome for individuals in each of the Mojac dilemmas, while communitarians made decisions that favoured the maintenance of group norms, values and laws. Dual-oriented respondents sometimes favoured the individuals and sometimes favoured the group in their final decisions. Analysis by anova found the differences between each group to be highly significant [F (2, 2901) = 897.39, P < 0.001].

Discussion

도덕지향은 최종 의사결정 문항을 유의미하게 예측했다.

This study has provided empirical evidence of the validity of the Mojac measure of moral orientation. In addition, the results of our research support the hypothesis that an individual's libertarian−communitarian moral orientation is a determinant of their moral decision making. In each of the samples tested, a person's moral orientation was found to be a significant predictor of their responses to the final moral decision questions embedded in the Mojac Scale.

연령, 성별, 문화에 따른 차이는 크지는 않았지만 유의했음. 나이를 들수록 communitarian이나 표집의 연령 폭 자체가 좁음.

Age, gender and cultural differences, although not large, were significant. Older respondents tended to be more communitarian; however, the distribution of respondents' ages in the samples is not representative of the general population. Some differences in LibCom scores were observed across 5 year-of-study student cohorts and were found to be weakly predicted by exposure to medical education rather than by age. However, the cross-sectional design of the study does not allow any inference concerning change in moral orientation. Longitudinal research exploring moral orientation change is required.

남성이 보다 libertarian, 여성이 보다 communitarian. 

Men generally were more libertarian and women more communitarian, although a notable exception was the predominantly female psychology student sample, which produced a significantly lower mean LibCom score (more libertarian) compared to the medical school samples. Respondents from the Israeli samples, coming from a somewhat more collectivistic culture, were generally more communitarian. While a near normal distribution of scores was observed within each group (men versus women and within each cultural group), indicating that the differences within groups were much greater than the differences between groups, further research regarding the influence of age, gender and culture is required. If differences are consistently found then the establishment of separate norms might be warranted.

도덕지향이 기존의 인성검사에서 확인되는 scale과 잘 일치하는 양상을 보임

Importantly, libertarian−communitarian moral orientation scores were found to be related to well validated personality scales in a conceptually coherent patterning. 

• High Mojac scorers (indicating an extreme communitarian moral orientation) had tendencies (as identified by parallel test instruments) to be authoritarian, conscientious, perfectionistic and self-controlled, while 
• low scorers (indicating an extreme libertarian orientation) tended to be disorderly, narcissistic, abstracted and unrestrained. 

Thus, when presented with an ethical dilemma in a medical situation, extreme communitarians might tend to be inflexible, reliant on procedures, rules and their perception of the ‘authority’ of medicine at the expense of the unique needs, rights and autonomy of their patients. Conversely, extreme libertarians might be overly flexible and ignore or bend the usual rules of procedure while being disproportionately concerned for the rights, well-being and liberty of patients and themselves as doctors.

Mojac Scale이 의료와 관계없는 딜레마 상황을 사용하여 이뤄졌지만, 다양한 맥락에 대한 일반화가 가능하다는 가정을 지지하는 많은 연구가 있음.

An important point concerns the use of non-medical dilemmas in the Mojac Scale (which is also the case with the well known tests of moral reasoning). The aim of such tests, and many others, is to measure individual differences in a particular psychological construct: in this case, moral orientation. The assumption is that these individual psychological differences influence a person's behaviour across situations. While this is arguable, the extensive literature on personality traits generally supports the assumption that traits generalise across different contexts. The correlations found between the Mojac moral orientation scores and the personality trait scores noted above empirically support the notion that, regardless of the stimulus used, the scale is measuring a psychological trait or tendency.

의과대학 학생선발에서의 활용. 인지적 척도에 대해서는 일정 점수 이상, 혹은 일정 석차 이상의 학생을 선발하게 된다. 그러나 도덕지향에 대해서는 극단의 성향을 보이는 지원자를 배제하는 것이 보다 합당하다. 2SD정도를 제안한다.

The use of tests in medical school selection procedures, be they tests of academic ability, cognitive skills, personality traits or moral orientation, requires that each test reliably measures the trait or ability it purports to. For ethical reasons, those charged with the responsibility for assessing and selecting medical school students clearly would need to consider the properties of any measure used. Additionally, considerable care needs to be taken in establishing how scores determine selection. Typically, as is the case with cognitive measures, test scores are ranked from highest to lowest and a ‘cut-point’ determined, above which applicants are retained in the selection pool. However, this might be an inappropriate procedure with tests that indicate individual differences in moral orientation, moral reasoning or moral values. To admit only high scoring applicants on such tests would require a test to produce a range of scores from the ‘most likely to be moral’ to the ‘least likely to be moral’ and the validity of such a test would be highly questionable. In view of the correlations found in the present study, an argument can be made that extreme high and low scorers, in this case extreme communitarians and extreme libertarians (perhaps defined by cut-points of + 2 SD and − 2 SD from the mean, respectively), could be considered for exclusion from the applicant pool on the grounds that their moral orientation is likely to be vocationally incongruent with the ethical standards and requirements of the medical context. The substantial majority who remain in the applicant pool would approximately equally value the needs, rights and well-being of individual patients and the needs, rights and well-being of others, the profession and society as a whole and so might be more likely to behave in an ethically appropriate way in the practice of medicine.

Screening out extreme scorers assumes, by definition, that a majority of applicants have the qualities to practise medicine ethically, particularly, as noted by Singer,2 if the medical education undertaken by successful applicants includes ethics training, evaluation of ethics in performance, and an ethical learning environment. Rather than select on the basis of high moral reasoning scores, it would seem more realistic and appropriate to screen out those few who indicate an extreme moral orientation. This would allow for moral development with time, education and experience and acknowledgement of the fact that most health professionals behave ethically. Most people are able to consider the needs and perspectives of both the individual and the group in their daily lives. If most did not, it is unlikely we humans would have survived and thrived as we have.

Selection of medical students according to their moral orientation.

의과대학 지원자의 '윤리성'을 평가하는 것이 가능할까?(Journal of Medical Ethics, 2001)

Is it possible to assess the “ethics” of medical school applicants?

Michael Lowe, Ian Kerridge, Miles Bore, Don Munro and David Powis Fiji School of Medicine, Fiji, and University of Newcastle, Australia

의과대학의 학생선발에서 지원자의 도덕성을 평가하는 어렵지만 중요한 일이다. 그러나 윤리지식, 도덕추론능력, 윤리적 신념 등을 평가하는 것은 부적절한데, 이런 것은 교육을 통해 개발될 수 있는 것이기 때문이다. 윤리적 이슈에 관한 태도와 윤리적 민감성은 인적특성에 대한 검사의 맥락에서 시험의 대상이 될 수도 있다. 모든 '윤리'시험은 입학에 적용되기 전에 validation이 필요하다. 

Questions surrounding the assessment of medical school applicants’ morality are difficult but they are nevertheless important for medical schools to consider. It is probably inappropriate to attempt to assess medical school applicants’ ethical knowledge, moral reasoning, or beliefs about ethical issues as these all may be developed during the process of education. Attitudes towards ethical issues and ethical sensitivity, however, might be tested in the context of testing for personality attributes. Before any “ethics” testing is introduced as part of screening for admission to medical school it would require validation.We suggest a number of ways in which this might be achieved. (Journal of Medical Ethics 2001;27:404–408)

UME나 GME가 장기적으로 윤리적 행동양상을 바꿀 수 있다는 근거가 부족한 상황에서, 비윤리적인 의사를 줄이는 길은 UME나 GME에 들어서기 이전에 사전에 차단하는 것이다. 

Given the paucity of evidence that undergraduate or postgraduate education may change or shape ethical practice in the long term, it seems that the only way to prevent people like Dr Shipman from continuing in the profession might lie in attempting to identify unethical doctors prior to entering medical school or during their undergraduate or postgraduate education, and excluding them from the profession before they cause harm.

윤리학은 우리가 어떻게 행동해야 하는가에 대한 학문이며, 비윤리적 의사란 하지 말아야 할 일을 하거나, 해야 하는 일을 하지 않는 의사이다. 

Ethics is the study of what we ought to do. An unethical doctor is therefore a doctor who does things that he or she ought not, or does not do the things that he or she should.

의사들을 윤리적으로 행동하게 만드는 요인에는 여러가지가 있다. 교육적 차원에서 이 요인들은 두 가지로 나뉘는데, 하나는 가르칠 수 있는 것이고 다른 하나는 타고나는 것이다.

There are a number of factors that enable doctors to act in an ethical way, including a desire or motivation to do so, a knowledge of ethical issues, the development of communication skills and other skills required for medical competence, a capacity for moral or ethical reasoning (we will use the terms “moral” and “ethical” synonymously), and an individual’s beliefs, attitudes, and sensitivity to ethical issues. Educationally, these factors appear to fall into two main groups—those that can be taught, and those that appear to be innate.

학생의 윤리적 추론능력을 평가해야하는가에 대한 또 다른 문제는, 윤리적 추론이 다른 종류의 추론과 유사해서 그 원리들을 도덕적인 문제에 적용시키기만 하면 되는 것이다. 다른 시험을 잘 보는 학생이 이 시험도 잘 볼 가능성이 높다.

A different problem occurs with the question as to whether we should examine students’ ethical reasoning. In many ways, ethical reasoning is like any other form of reasoning, it is simply the application of logic to matters of morality. Students who are selected for medicine on the basis of other tests of logic, are likely to do well at tests of moral reasoning as well.

그러나 '도덕적 추론'이라는 용어는 조금 다른 의미를 갖는데, 단순히 그 논리적 절차 뿐만 아니라 콜버그의 도덕성 발달이론에서 언급되는 것과 마찬가지로 윤리적 성숙, 윤리적 발달 과정이라는 것이다. 콜버그는 여섯 개의 단계를 거쳐 도덕성 발달이 이뤄진다고 결론지었다. 이 이론을 바탕으로 MJI, DIT, SMR 등의 도덕적 추론의 척도가 개발되었다.

However, the term “moral reasoning” also has a slightly different meaning, referring not only to a process of logic, but also to a process of ethical maturation or development, such as in Kohlberg’s theory of ethical development. Kohlberg came to the conclusion that moral development occurred in six defined stages, leading from a state of moral immaturity in which ethical decisions were taken ad hoc, to higher levels of moral development which involved individuals acting objectively, rationally, and impartially, following universal ethical principles of a higher morality. His theory has been studied extensively, and it underlies the development of measures of moral reasoning such as the Moral Judgement Interview (MJI),3 The Defining Issues Test (DIT),4 and the Sociomoral Reflection Measure (SRM).5

콜버그의 이론은 나이를 먹을 수록 도덕적으로 추론하는 인지능력이 상위 단계로 올라간다고 보았으나, 나이는 도덕성 발달의 유일한 변인이 아니다. 도덕적 의사결정을 내려야 하는 상황을 다양하게 경험해볼수록 도덕 추론 점수도 높아지는 것으로 연구되고 있다. 또한 일부 연구들은 교육을 통해서 도덕추론점수를 높일 수 있음을 밝힌 바 있다.

Kohlberg’s theory suggests that as the individual ages, the cognitive ability to reason morally moves through a hierarchy of invariant stages.3 Age, however, is not the only variable in moral development, as studies suggest that the opportunity to experience an enriched moral decision making environment may also influence moral reasoning scores.6 Self, Baldwin, and Wolinsky provided an example of this effect when they demonstrated that medical students had a highly significant gain in the adoption of principled reasoning as measured by the DIT after a course in medical ethics.7 This has also been observed in other longitudinal studies,8 and in a meta-analysis by Schlaefli, Rest, and Thomas. 9 These findings support the notion that educational experience can increase moral reasoning scores.

그러나 도덕적 추론이 도덕적 의사결정으로 연결되는지는 불분명한데, 이는 도덕적 추론능력에 대한 검사가 도덕적 의사결정을 내리고 난 뒤의 합리화 과정에 근거하기 때문이다. 실제로 콜버그의 이론도 도덕적 추론에 대해서 도덕적 추론이 도덕적 의사결정과는 다르다고 전제하고 있다.

However, it is unclear how moral reasoning is related to moral decision making, since tests of moral reasoning tend to be based upon the justifications produced by an individual after a moral decision has been made. Indeed, Kohlberg’s theoretical premise in developing his theory of moral reasoning was that reasoning is independent of moral decisions made.

따라서 도덕적 추론이 교육적 경험에 따라서 바뀔 수 있는 것이고, 도덕적 의사결정과 무관할 수 있기 때문에 도덕적 추론에 대한 척도는 의과대학 학생선발에 활용하기는 부적절하다.

It appears therefore that, since moral reasoning has been shown to change with the educative experience and may be unrelated to the moral decisions individuals make, measures of moral reasoning are unlikely to be suitable for inclusion in the selection of applicants for medical education.

윤리적 신념, 태도, 민감성 Ethical beliefs, attitudes and sensitivity

지원자의 윤리적 신념을 검사하는 것은 기술적으로 어렵지는 않으며, 어떤 윤리적 신념은 일부 의료환경에서는 진료를 어렵게 만들기도 한다. (여성의 할례, 동물에 대한 학대)

It would not be technically difficult to examine applicants for particular ethical beliefs, and there are some ethical beliefs that may make medicine difficult to practise in some environments. For example, applicants from some ethnic groups may believe that it is reasonable to perform female circumcisions despite this being widely considered in Western society to be immoral; some applicants may be willing to sabotage animal experiments out of interest for the animals; yet others may believe it is a valuable aim of humanity to pharmacologically enhance sportsmen and women so they can perform better.

그러나 우리는 나중에 교육을 통해서 개발되거나 바뀔 수 있는 윤리적 신념때문에 지원자들이 의과대학에 불합격하는 것을 바라지 않는다. 신입생에게는 윤리적 문제에 대한 단순한 신념 정도만 있으면 되고, 그 지평을 넓혀주는 것이 의학교육의 역할이다.

However,we do not believe applicants to medical school should be rejected because of their individual ethical beliefs, as ideas can be developed or discarded by individuals throughout their medical training and later careers. Unsophisticated beliefs about ethical subjects should be expected in junior students, and one role of medical education is to broaden their experience and knowledge-base.

의과대학 지원자에 대하여 사례를 통한 윤리적 민감성을 평가하고, 과제를 줄 수도 있다. 예컨대 예상가능한 윤리적 이슈라든가, 다양한 행동에 대한 이유를 합리화하는 것 등이다.

It may be possible to test for ethical sensitivity by providing applicants with a vignette and giving them a task to perform, such as coming up with a list of ethical issues that might arise, or justifying the various courses of action.

비윤리적 의사의 모습은 - 자기애적 자기중심적이고, 자신의 이익밖에 고려하지 않는 - 의료전문직을 아는 사람이라면 누구나 친숙한 모습일 것이다. 

This picture of unethical doctors—as narcissistic egotists, unconcerned with anyone’s interests but their own—is familiar to anyone involved with the medical profession or its representations in the popular press; and these descriptions match the profiles described in some other studies of unethical doctors.

윤리학 vs 정신과학적 진단 Ethics versus psychiatric diagnosis

비윤리적 행동과 관련될 수 있는 다양한 인적특성을 다뤘지만, 극단적 경우에는 이런 인적특성이 인격장애처럼 보일 수 있다. 

We have described a number of personality characteristics that may be associated with unethical behaviour. In the extreme cases, some of these personality traits may even be described as personality disorders, although clearly this does not apply in all cases.

일부 연구자들은 'bad'와 'mad'사이의 관계를 지적한 바도 있지만, 이 두 가지 영역이 가능한 서로 멀리 떨어져있어야 한다고 느끼는 듯 하다. 왜냐하면 정신과학적 진단은 도덕적 판단과 구분되어야 하기 때문이다. 그러나 정신과학적 진단은, 특히 인격장애에 있어서는 도덕적 판단의 범위까지 넘어오기도 하고 겹치는 부분이 있기도 하다.

Several authors have commented upon the links between the ethical domain (“bad”) and the psychiatric domain (“mad”).13 Many commentators feel these domains should be kept apart as far as possible, and that psychiatric diagnosis should be kept separate from moral judgments.14 Yet psychiatry has always had a tendency to move beyond its brief, and moral judgments and psychiatric diagnoses often appear to overlap, particularly in the area of personality disorders.

지원자의 인격장애를 검사하려면, 대부분의 의과대학이 높은 학업지능을 기반으로 학생을 선발하는 것을 인식할 필요가 있다. 반사회적 성향과 싸이코패쓰에 관한 연구를 보면 이러한 사람들을 "정상인, 심지어는 매력적, 매혹적인 외형의 사람들과 구분하는 것"이 얼마나 어려운지를 보여준다. "반사회적 인격장애를 가진 사람들은 진실을 말하지 않으며, 일반적 도덕성 잣대에 따라서 임무를 수행한다."

If we are going to test applicants for personality disorders, it is important to realise that most medical school applicants have already been picked on the basis of a high level of academic intelligence. The literature on antisocial personalities and psychopaths emphasises how difficult it is to diagnose this condition as people with these conditions may present with “a normal and even a charming and ingratiating exterior . . .. Antisocial personality disorder patients do not tell the truth and cannot be trusted to carry out any task or adhere to any conventional standard of morality.”15

자기애적 성향은 그 반대인데, 잡아내기가 그다지 어렵지 않다. 그리고 이러한 사람을 진단하는 많은 도구가 있다. 그러나 이렇나 사람들을 걸러내는 것의 문제는 비록 이들이 함께 일하기에 그다지 즐거운 사람들은 아니지만, 어느 분야의 선구자에게 이러한 특징들이 심심찮게 발견된다는 사실이다. 만약 자기애적 성향을 가진 사람들을 다 걸러낸다면, 모든 사람이 꺼려하는 분야에서도 자신의 신념을 고집해서 새로운 아이디어를 시도할 줄 아는 소중한 미래의 지도자를 잃는것은 아닐까?

The narcissistic personality on the other hand, appears to be relatively easy to trap in his or her own conceits, and there are a number of instruments used for diagnosing this condition. The difficulty with excluding people with this type of personality is that, although it is widely agreed that they are unpleasant to work with, we are struck by the prevalence of narcissistic traits among leaders of the profession. If we reject the narcissists, do we lose valuable future leaders who through their own egotism try new ideas and procedures that others do not dare?

결론 Conclusion

Medical school entry is based upon a number of factors. Cut off marks for academic performance are perhaps the most popular methods of excluding potential applicants, although there is no evidence to justify the extremely high marks required for many courses. Courses are now including tests of logical reasoning, tests of lateral thinking, and testing that is known to discriminate in favour of certain groups (eg women) to the disadvantage of others.

여태껏 의과대학 지원자의 도덕적, 윤리적 특성 명확히 드러내는 검사는 없었다. 

We are not aware of any medical schools which test explicitly for moral or ethical attributes of applicants for medicine, although these topics are frequently covered in interviews. The reason for this is probably concerns about the methodological issues involved in defining and testing ethical attributes, and fear of introducing new biases and new forms of unjustified discrimination into the selection process. Indeed, if we are to develop measures for assessing applicants’ attitudes and sensitivity to moral issues, it is important that these should not be based purely on theoretical structures, but that they also be validated empirically. The main difficulty with validation is how to define unethical behaviour well enough to test any measures developed.

Is it possible to assess the "ethics" of medical school applicants?

PMID:

 

11731605

 

[PubMed - indexed for MEDLINE] 

PMCID:

 

PMC1733480

 

Free PMC Article

입학전형과 학업능력: 세 가지 입학전형 비교(Medical Education, 2014)

Selection and study performance: comparing three admission processes within one medical school

Nienke R Schripsema,1,2 Anke M van Trigt,2 Jan C C Borleffs2 & Janke Cohen-Schotanus1,2

OBJECTIVES:

본 연구의 목적은 (1)고등학교 성적, 자발적 다면선발, 추첨이라는 세 가지 방법으로 입학한 학생들의 학업능력을 비교하고, (2)다면선발을 통해서 입학한 학생이 여기서 떨어진 학생보다 우수한지를 알아보고 (3)다면선발에 지원했다는 것 자체가 학업능력과 연관되는가를 보고자 한다.

This study was conducted to: (i) analyse whether students admitted to one medical school based on top pre-university grades, a voluntary multifaceted selection process, or lottery, respectively, differed in study performance; (ii) examine whether students who were accepted in the multifaceted selection process outperformed their rejected peers, and (iii) analyse whether participation in the multifaceted selection procedure was related to performance.

METHODS:

지식시험, 프로페셔널리즘 점수, 학습발달, 중퇴의 네 가지를 비교하였다. 추첨입학 학생군을 다면선발에 지원한 학생과 그렇지 않은 학생으로 나눴다. ANCOVA, 로지스틱 회귀분석, Bonferroni 검정을 사용하였다.

We examined knowledge test and professionalism scores, study progress and dropout in three cohorts of medical students admitted to the University of Groningen, the Netherlands in 2009, 2010 and 2011 (n = 1055). We divided the lottery-admitted group into, respectively, students who had not participated and students who had been rejected in the multifaceted selection process. We used ancova modelling, logistic regression and Bonferroni post hoc multiple-comparison tests and controlled for gender and cohort.

RESULTS:

고등학교에서 최고수준 성적을 받은 그룹이 지식검사에서 높은 점수를 받았으며, 1학년때 더 많은 학점을 이수하였다. 또한 이 그룹은 다면선발에 지원하지 않은 추첨입학 학생군에 비해서 최상위권 프로페셔널리즘 점수를 더 많이 받았다. 다면선발로 입학한 학생은 다면선발에 지원하지 않은 추첨선발 학생보다 지필고사 점수가 높았으며, 두 가지 추첨선발 학생군보다도 프로페셔널리즘 점수를 잘 받았다. 다면선발에 지원하지 않았던 추첨선발 학생군은 다른 모든 그룹보다 1학년과 2학년 이수학점이 적었고, 중퇴율은 학생군별로 달랐지만, 보정 후에는 유의하지 않았다.

The top pre-university grade group achieved higher knowledge test scores and more Year 1 course credits than all other groups (p < 0.05). This group received the highest possible professionalism score more often than the lottery-admitted group that had not participated in the multifaceted selection process (p < 0.05). The group of students accepted in the multifaceted selection process obtained higher written test scores than the lottery-admitted group that had not participated (p < 0.05) and achieved the highest possible professionalism score more often than both lottery-admitted groups. The lottery-admitted group that had not participated in the multifaceted selection process earned fewer Year 1 and 2 course credits than all other groups (p < 0.05). Dropout rates differed among the groups (p < 0.05), but correction for multiple comparisons rendered all pairwise differences non-significant.

CONCLUSIONS:

고등학교에서 최상위 GPA를 받은 것이 학업능력을 가장 잘 예측하였다. 비학업적 수행능력에 있어서는 다면선발이 가장 효율적인 방식인 것으로 보인다. 다면선발에 지원하여 참여하는 것 자체가 높은 수행능력의 예측인자였다. 

A top pre-university grade point average was the best predictor of performance. For so-called non-academic performance, the multifaceted selection process was efficient in identifying applicants with suitable skills. Participation in the multifaceted selection procedure seems to be predictive of higher performance. Further research is needed to assess whether our results are generalisable to other medical schools.

입학점수와 입학 후 수행능력의 관계에 대한 연구에서 두 가지 한계가 있다. 첫째는 합격자만을 대상으로 하기에 restriction of range가 생긴다는 것이고, 둘째는 단일한 전형으로 학생을 선발하는 많은 학교에서는 서로 다른 입학전형간 비교가 어렵다는 점이다.

Studies on the relationships between admission scores and later performance, however, are characterised by two limitations. Firstly, most study samples have a restriction of range because performance data are only available for applicants who were accepted in the selection process. It remains unclear how rejected applicants would have performed if they had been admitted. Secondly, in most medical schools, all students are admitted through the same admission process based on a single set of criteria, which limits opportunities to compare different admission processes.

합격자와 불합격자의 비교연구

In the literature, few studies investigating performance differences between accepted and rejected applicants have been reported.8,9

서로 다른 입학전형간 비교

A few studies have addressed the effects of different admission processes on student performance within one medical school.

네덜란드의 3단계 의과대학 학생선발 시스템

The Netherlands has a three-step admission system. In the first step, applicants with a top pre-university GPA are offered admission. Participation in the second step is voluntary. In this step, students participate in a multifaceted selection process that is organised by each medical school separately. The third step is a national weighted lottery, in which applicants who were rejected in the second step as well as applicants who have not participated in the second step can enrol.

METHODS

Context

네덜란드의 그로닝겐 의과대학. 3년의 BA와 3년의 MA. 교육과정 개괄

This study was performed at the University of Groningen in the Netherlands. The problem-based curriculum consists of a 3-year pre-clinical Bachelor’s and a 3-year clinical Master’s programme. The first year of the Bachelor’s programme includes four 10- week blocks, a year-long professionalism course, and the inter-university progress test.15,16 Each part provides students with a fixed number of course credits under the European Credit Transfer System. One course credit equals 28 hours of study. The maximum number of course credits per year is 60.

Admission processes in the Netherlands

네덜란드 학생선발 단계

The Netherlands has a national policy for medical school admissions according to which applicants can be admitted through one of three steps. 

• In the first step, students with a pre-university GPA of ≥ 8 (on a scale ranging from 1 = poor to 10 = excellent) gain admission to the medical school of their choice without further assessment. This grade average is calculated by averaging each applicant’s mean grade on pre-university school examinations and mean grade on national final examinations. As only approximately 4% of all pre-university graduates achieve a GPA of ≥ 8, this grade indicates excellent achievement. 
• In the second step, applicants can be accepted in a multifaceted selection process. Each medical school organises its own selection process in which participation is voluntary. Selection processes differ among medical schools, but they usually consist of two rounds of assessments in which various knowledge-based and behavioural variables are measured. 
• In the third step, applicants are admitted through the national weighted lottery. In the lottery, applicants are categorised based on their pre-university GPA. Four categories of GPA are distinguished: 7.5–7.9; 7.0–7.4; 6.5–6.9, and < 6.5. The ratio of applicants admitted by category is 9 : 6 : 4 : 3.11,17 Applicants who were rejected in the multifaceted selection process can be admitted through the lottery. Until 2011, approximately 50% of the places available at each medical school were assigned through the national weighted lottery.

네덜란드에서는 매년 8500명이 지원하여 2780개의 정원을 놓고 경쟁함. 410명이 그로닝겐 의과대학에 배정됨. 

Every year, around 8500 applicants apply to medical school in the Netherlands. There are 2780 places available, 410 of which are at the University of Groningen. Approximately 60 of these places are allocated to the International Bachelor of Medicine programme. The remaining 350 are allocated to the Dutch Bachelor of Medicine programme. In the current study, we included only students enrolled in the Dutch Bachelor of Medicine programme.

Participants

연구대상자

We included all 1055 students admitted to the Dutch Bachelor of Medicine programme at the University of Groningen in 2009, 2010 and 2011 (69% female; mean age at the start of Year 1: 18.6 years; mean pre-university GPA: 7.3). We defined four groups of students: 

• (i) students who were admitted based on a pre-university GPA of ≥ 8 out of 10 (n = 143; 71% female; mean age at the start of Year 1: 18.0 years; mean pre-university GPA: 8.2); 
• (ii) students who were accepted in the multifaceted selection process (n = 295; 74% female; mean age at the start of Year 1: 18.5 years; mean pre-university GPA: 7.1); 
• (iii) lottery-admitted students who had been rejected previously in the multifaceted selection process (n = 315; 69% female; mean age at the start of Year 1: 18.5 years; mean pre-university GPA: 7.1), and 
• (iv) lottery-admitted students who had not participated in the selection process (n = 302; 63% female; mean age at the start of Year 1: 19.1 years; mean pre-university GPA: 7.0). Descriptive statistics on percentages of women, mean age and pre-university GPA within the four groups are depicted in Table 1.

The privacy policy of the University of Groningen states that student records can be used for research purposes as long as reports cannot be traced back to individual students.18 In accordance with this privacy policy, anonymised data were derived fromthe university administration.

다면선발 절차 Multifaceted selection process

1단계와 2단계가 있음.

In 2009, 2010 and 2011, the University of Groningen selection process consisted of two rounds.

1단계

In the first round, applicants were asked to send in a pre-structured written portfolio, based on the procedure developed by Erasmus MC medical school,11 with an additional section on reflection. 

• The portfolio contained three sections which covered, respectively, pre-university education, extracurricular activities and reflection. 
• In the section on extracurricular activities, points were granted when applicants were able to show an ability to participate in multiple activities at the same time (i.e. to combine pre-university education with additional activities). Extracurricular activities yielded points if they met fixed criteria on total duration and the amount of time spent on the activity per week. Only activities that had been carried out in the preceding 1.5 years, for at least 5 months consecutively, for more than 3.5 hours per week, yielded points. 최근 1년6개월 내에 5개월 이상 연속적으로 매주 3.5시간 이상 한 활동만을 인정함
• For the section on reflection, applicants were asked to carry out a number of reflective assignments. For example, applicants were required to ask three people in their network to give two reasons why the medical profession might suit them and one reason why they might be better off not practising medicine. Applicants were asked to reflect on these statements in a short essay. In the evaluation process, only the reflection was assessed. Evaluations of the selection process indicated that applicants required 40–60 hours to complete the first-round portfolio. 한 가지 사례는 3명의 사람에게 지원자가 의과대학에 적절한 이유와 그렇지 않은 이유에 대해 받은 뒤, 이것에 대한 짧은 에세이 기술

2단계: 225명

The 225 applicants who scored highest in the first round were invited to participate in the second round of the multifaceted selection process, which lasted an entire day and took place at the University of Groningen Medical School. 

The day was divided into four blocks comprising, respectively, a writing assignment, a patient lecture with subsequent assignments, a scientific reasoning block, and a series of short interviews and role-plays. 

• For the writing assignment, applicants were asked to write an essay about a societal problem for which ethical decision making was key. For example, in 2009, applicants were asked to write an essay on China’s one-child policy. To help them prepare for this assignment, applicants were sent a package with information about the subject a week before the second round. 
• The lecture was focused on medical knowledge, ethical dilemmas and professional behaviour, and consisted of a presentation with integrated videos, similar to the format of a video-based situational judgement test.19,20 Applicants were required to answer questions about the content of the lecture, describe and analyse the presented ethical issues and recognise (un)professional behaviour. 
• In the scientific reasoning block, applicants were asked to read a scientific article about which they were then required to answer questions that tested analytic, creative and practical skills.21,22 
• The fourth block consisted of an MMI like 5,23 series of short assignments in which applicants were asked to reflect on the assignments in their first-round portfolio, carry out a role-play that focused on communication skills, and a three-phase role-play in which they collaborated with two fellow applicants. In this scenario, three applicants first had to prepare a problematic interaction with an actor. They then held the actual conversation, after which they were asked to reflect on the course of the conversation.

Scores on the four blocks were calculated and applicants were ranked based on their total scores. The fourth block was given a double weighting. 

통계분석 Statistical analysis

We performed analysis of covariance (ANCOVA) to examine group differences in written test grades and in the number of course credits students earned in their first, second and third years. To determine which groups differed, we performed Bonferroni post hoc multiple-comparison tests for the corrected means. We conducted logistic regression analyses with changing reference groups to examine group differences in the percentages of students who achieved the highest possible score in the professionalism course. Students who dropped out of medical school in the first half of each year were excluded from these analyses because they were unable to earn course credits in the entire period under analysis and their inclusion might have caused us to overestimate effects on study progress. To assess group differences in dropout rates, we conducted logistic regression analysis including the entire group (n = 1055). All analyses were performed using IBM SPSS Statistics for Windows Version 20.0 (IBM Corp., Armonk, NY, USA) and controlled for gender, age and cohort. After initial analyses, age was eliminated as a covariate as it was not a significant predictor in any of the models. We did not correct for pre-university GPA because the group of students who were admitted based on a top pre-university GPA was defined by this variable and the other groups did not differ significantly.

기존 연구에서 중퇴율, 성적, 임상실습성적 등에서 추첨선발 학생군(4군)의 학업능력이 낮게 보고된 것은 추첨선발 학생군 전체(3군+4군)이 아니라 4군에 의한 영향일 가능성이 높아 보인다. 우리의 연구 결과에 따르면 지식검사와 학습발전정도에 있어서는 다면선발이 좀 더 예측력을 갖는 것으로 보인다. 이러한 결과를 종합하면 의과대학은 지원절차에 시간과 노력을 쏟은 지원자만을 선발하는 것이 도움이 될지도 모른다. 이전에 어떤 전형에도 지원한 적이 없는 추첨입학학생군의 낮은 학업능력은 다면선발에 내재된 '자발성'에 의해 설명될 수 있다.이는 '자기선발'과정이기도 하다. 동기부여가 더 잘 된 학생은 의과대학 입학을 위한 모든 가능한 기회를 다 사용해볼 것이며, 동기부여가 덜 된 학생은 그냥 추첨선발까지 기다려볼 것이다.

The lower performance of non-participants might indicate that previously reported differences in dropout rates,11 test grades28 and clerkship grades12 between selected and lottery-admitted students in the Dutch system can be attributed to lottery- admitted applicants who have not participated in the selection process, rather than to the lottery-admitted group as a whole. Our results in fact indicate that for knowledge test scores and study progress, participation in the multifaceted selection process may be more predictive than acceptance in this process. As such, it might be profitable for medical schools to admit only applicants who have put time and effort into their applications. The lower performance of non-participants may be explained by the voluntary nature of the multifaceted selection process, which implies a mechanism of self-selection that is induced by the Dutch admissions system.29 Highly motivated students might use every opportunity to achieve admission into medical school, whereas less motivated students might choose to wait for the lottery.

이러한 동기부여의 차이에 대하여 가능한 설명은 선발절차에 참여(participation)하는 것에도 40~60시간의 노력이 필요하다는 것이다. 동기부여가 잘 된 학생은 이러한 것에 쉽게 노력과 시간을 투자할 수 있는 반면 그렇지 않은 학생은 이 자체가 힘들 수 있다.

A possible explanation for this difference in motivation is that participation in the selection process requires 40–60 hours of work. Highly motivated students might easily have invested this kind of time and effort, whereas less motivated students may have perceived this threshold as too high.

Med Educ. 2014 Dec;48(12):1201-10. doi: 10.1111/medu.12537.

Selection and study performance: comparing three admission processes within one medical school.

의과대학 학생선발 과정에 사용되는 자료들과 중요도의 변천사: 1980년대 이후로 무슨 일이 있었나? (Academic Medicine, 2013)

An Overview of the Medical School Admission Process and Use of Applicant Data in Decision Making: What Has Changed Since the 1980s?

Alicia Monroe, MD, Erin Quinn, PhD, MEd, Wayne Samuelson, MD, Dana M. Dunleavy, PhD, and Keith W. Dowd, MA

PURPOSE:

현재 미국/캐나다 의과대학의 학생선발 절차를 알아보고, 1986년과 어떻게 다른지 비교한다.

To investigate current medical school admission processes and whether they differ from those in 1986 when they were last reviewed by the Association of American Medical Colleges (AAMC).

METHOD:

2008년 MCAT점수를 활용하는 모든 미국과 캐나다의 의과대학을 대상으로 학생선발 절차에 관한 설문을 시행하였다.

In spring 2008, admission deans from all MD-granting U.S. and Canadian medical schools using the Medical College Admission Test (MCAT) were invited to complete an online survey that asked participants to describe their institution's admission process and to report the use and rate the importance of applicant data in making decisions at each stage.

RESULTS:

120개 대학이 응답하였다. 면접을 통해서 인적특성을 평가한다고 대답하였으며, 1986년과 비교하였을 때 1단계(면접대상자 선발 평가)에서 학업 성적관련 자료의 사용이 늘었다. 1986년에는 GPA가 거의 모든 단계의 합격자 결정에 있어서 가장 중요했던 반면, 2008년에는 1단계, 2단계에 따라서 중요시되는 자료가 서로 달랐다. MCAT과 학부 GPA는 1단계에서 중요했으며, 면접과 추천서 등이 최종적으로 누가 합격할지를 결정하는데 중요했다.

The 120 responding admission officers reported using a variety of data to make decisions. Most indicated using interviews to assess applicants' personal characteristics. Compared with 1986, there was an increase in the emphasis placed on academic data during pre-interview screening. While GPA data were among the most important data in decision making at all stages in 1986, data use and importance varied by the stage of the process in 2008: MCAT scores and undergraduate GPAs were rated as the most important data for deciding whom to invite to submit secondary applications and interview, whereas interview recommendations and letters of recommendation were rated as the most important data in deciding whom to accept.

CONCLUSIONS:

의과대학 학생선발절차의 복잡성을 보여주는 연구이며, 지원자의 댜양한 측면을 총체적으로 평가하는 holistic approach의 사용이 늘었음을 보여준다. 

This study underscores the complexity of the medical school admission process and suggests increased use of a holistic approach that considers the whole applicant when making admission decisions. Findings will inform AAMC initiatives focused on transforming admission processes.

학생선발의 여러 절차에는 어떤 순환적 구조가 있다. 의과대학이 변하면 입학정책을 통해서 지원자 풀에 영향을 주고, 입학결정이 이뤄지는 법적, 사회적 맥락이 있으며, 의과대학 교육과정이 변한다.

We suggest that they have a cyclical relationship with the admission process: Changes in academic medicine affect the admission process through their influence on the applicant pool, the legal and social contexts in which admission decisions are made, and the medical school curriculum.

1980년대에는 의과대학 지원자 풀이 작아져서 지원자의 수준이 떨어질 것이라는 우려가 있었다. 또한 소수자와 여성지원자 비율이 낮았다. 그러나 2008년 지원자 풀이 넓어졌으며 여기에는 아시아계 학생과 여학생의 영향이 크다.

Since the mid-1980s, the number and composition of medical school applicants have changed dramatically. At that time, there were concerns about a declining applicant pool and a potential decline in the academic quality of applicants.5 Additionally, the percentages of minority and female applicants were relatively low.6 By 2008, the applicant pool had grown and become more diverse with respect to Asian and female applicants.*

지원자의 수와 구성이 변한 것이 입학정책에도 영향을 주었다. (1)평가의 단계가 더 늘어났고, (2)좀더 정량화가능한 자료를 많이 사용하게 되었으며 (3)양적 자료와 질적 자료(학업적-비학업적 자료)를 모두 사용하게 되었다.

We suggest that these changes in the size and composition of the applicant pool may have affected the admission process in several ways when compared with that of the mid-1980s. 

• Given the increase in applicants, admission committees—especially those with large applicant pools—may add stages to the process to reduce the number of applicants remaining at each stage. 
• Second, with more applicants in the pool, admission officers may rely more heavily on data that are quantifiable and easily incorporated into pre-interview screening tools. 
• Third, in light of the changes in composition of the applicant pool, admission committees may use a combination of quantitative and qualitative (academic and nonacademic) data in order to achieve broad diversity in the student body.

법적 환경도 영향을 많이 주는요인이다.

The legal context in which admission committees operate has changed substantially, however. For example, the Supreme Court’s 2003 decision in Grutter v. Bollinger13 affirmed the importance of mission-driven, evidence-based admission decisions and introduced the concept of educational benefits of diversity.† It also established that all applicants must be considered through the same admission process, which allowed schools to change their diversity and admission policies. In 1986, only 28% of U.S. medical schools included diversity as a primary goal of their admission process,‡ whereas 57% did in 2008.14,15

법적, 사회적 환경의 변화는 적어도 다음의 두 가지를 통헤 입학에 영향을 준다. (1)1980년대보다 학생선발 과정에서 지원자에 대한 더 많은, 다양한 자료를 활용하게 되엇다. (2)지원자의 민족/인종, 성별. 사회경제적수준 등에 대한 고려가 많아졌다.

We suggest that, together, these legal and social context changes may have affected the admission process in at least two ways. 

• First, admission committees may now consider more and varied information about applicants in making admission decisions than they did in the mid-1980s. 
• Second, with a slightly more diverse applicant pool and a more permissive legal environment, admission committees may now be more likely to consider information about applicants’ race/ethnicity, gender, and/or SES background in the admission process.

1990년대와 2000년대에 의과대학에 많은 변화가 있었음.

In the 1990s and 2000s, a series of structural modifications to the medical school and residency accreditation processes, as well as new curricular resources, paved the way for fundamental changes in medical education. For example, the Liaison Committee on Medical Education18 (LCME) and the Accreditation Council for Graduate Medical Education19 revised their accreditation standards to require medical schools and residency programs to teach and assess professional attributes. LCME standard MS-31-A states, “A medical education program must ensure that its learning environment promotes the development of explicit and appropriate professional attributes in its medical students (attitudes, behaviors, and identity).”18 In addition, the AAMC Medical School Objectives Project series20 and the Institute of Medicine (IOM) report on behavioral and social sciences in medical school curricula21 identified— and, importantly, provided curriculum materials to help medical schools modify their curricula to teach—the broad knowledge, skills, and attitudes that graduating medical students should possess.

The admission process in 2008

Slightly more than half (57%, 68/120) of the respondents reported that their medical school’s admission decisions are made using a two-stage process that includes an initial application and an interview.

Slightly less than half (43%, 52/120) of the respondents reported that their schools use a three-stage process to make admission decisions, the same as reported by Mitchell4 in 1986 (43%, 49/113).

Overall, admission officers rated a wide range of data as important to admission committees’ decisions about which applicants to invite to submit secondary applications, interview, and accept into medical school (Table 1). However, the uses and importance of these data differed by the stage of the process.

Table 2 compares the relative importance of 15 types of data in making acceptance decisions in 1986 and 2008. Among the most important types of data, 64% (7/11) were nonacademic in 2008 compared with 50% (5/10) in 1986,4 suggesting that nonacademic data are more important to admission decisions today than in the past.

The admission interview in 2008

All responding admission officers reported that their medical schools conduct admission interviews.

Interviews were described as one-on one by 83% (99/119) of the responding admission officers. Most respondents (87%, 104/120) reported that interviews are conducted by admission committee members, whereas 17% (20/120) reported that they are conducted by staff and 68% (81/120) indicated that, in some cases, they are conducted by medical students.

Results showed that the admission interview is somewhat structured. The majority of respondents (65%, 77/119) indicated that interviewers are given general guidance about the content of the questions they should ask. 

Admission officials indicated that interviews are most often used to assess nonacademic characteristics and skills: Over 85% (more than 100 of 119) reported that interviews include questions about applicants’ motivation for pursuing a medical career, compassion and empathy, personal maturity, oral communication skills, service orientation, and professionalism.

학부 GPA와 MCAT점수가 중요함을 보여준다. 그러나 이것만이 전부가 아니라는 것을 보여주는 결과는, UGPA와 MCAT이 모두 최상위권인 1233명의 지원자 중에서조차 9%의 학생은 모든 의과대학에 불합격했다는 점이다. 

Chart 1 shows that although UGPAs and MCAT scores are important factors in admission processes, they are not the sole determinants of acceptance decisions. For example, 105 (9%) of the 1,233 applicants with UGPAs of 3.80 to 4.00 and MCAT total scores of 39 to 45 were not accepted by any of the medical schools to which they applied in 2008–2010. In contrast, 597 (18%) of the 3,324 applicants with UGPAs of 3.20 to 3.39 and MCAT scores of 24 to 26 were accepted by at least one medical school. These findings buttress the importance ratings data presented earlier, suggesting that a wide variety of data are important to admission decisions.

변하지 않은 점 Aspects of the admission process that have not changed

변하지 않은 점은 다음과 같다. (1)다양한 자료를 활용하고 있다. (2)두 단계 혹은 세 단계로 구성되어 있다. (3)MCAT점수를 중요하게 보고 있으며, MCAT과 학부GPA를 서로 비교하여 활용한다. (4)면접의 수, 길이, 형식이 거의 비슷하다. (5)인구통계학적 특성의 중요성은 낮다고 응답했다.

Certain aspects of the admission process are largely unchanged since the mid- 1980s. 

• First, as in 1986,4 admission officers today use a variety of data in making decisions, which suggests that they remain committed to evaluating both academic and nonacademic information.
•  Second, our data suggest that, as in 1986, schools’ admission processes are structured into two or three stages.
•  Third, in both 1986 and 2008, admission officers rated MCAT scores as important to each stage in the process and indicated that they use MCAT scores and UGPAs to provide an interpretive context for each another. 
• Fourth, the number, length, and format of admission interviews are the same as those described by Johnson and Edwards3 in 1991. Similarly, the admission interview continues to be the primary source of information about applicants’ personal characteristics.
• Finally, as in the 1986 survey,4 admission officers in our survey rated the importance of demographic characteristics in the admission process as relatively low

일부 의과대학의 입학위원회에서는 holistic review를 하는 것 자체가 소수인종이나 농촌지역 지원자들이 자신의 잠재력을 충분히 보여줄 수 있는 것이라고 생각하여, 명시적으로 인구통계학적 변인들을 고려할 필요를 낮추는 것으로 보인다.

We suggest, on the basis of the data presented in this article and comments made by focus group participants during site visits, that some medical school admission committees may feel that conducting holistic reviews allows URM and rural applicants to show their full potential and precludes the need to consider demographic variables explicitly.

변한 점 Aspects of the admission process that have changed

1980년대와 마찬가지로 두 단계, 세 단계의 과정을 거치나 각 단계에서 중요하게 보는 것이 달라진다.

As was the case in the mid-1980s, most admission committees use a multistage process to make decisions. However, our data suggest that admission committees now place different emphasis on applicant data at each stage of the process. For example, in summarizing the results of the 1986 survey, Mitchell4 noted that although test scores decreased in importance as decision making proceeded, importance ratings did not differ appreciably across the stages of the admission process. In contrast, our data suggest that admission committees now consider slightly different data when deciding whom to invite to submit secondary applications, interview, and accept. Academic data seem to be slightly more important in deciding which applicants to invite to submit secondary applications and to interview than in deciding whom to accept.

This difference is likely due to the increasing size of applicant pools and the ease of incorporating academic data into automated screening processes

However, we interpret the 2008 survey data reported in this article—and the qualitative data from the 2008 medical school site visits—as indicating that the inclusion of multiple stages does not preclude the use of a holistic admission process.

비학업적 자료에 대한 중요도가 높아진 것 역시 중요한 변화 중 하나이다.

Arguably, the most notable change in the admission process is the increased importance placed on nonacademic data in making acceptance decisions. In 2008, admissions officers rated more nonacademic data as “of high importance” than did admission officers in 1986. Further, all types of academic data dropped in ratings of relative importance in the 2008 survey compared with the 1986 survey (except MCAT scores), whereas nonacademic data such as interview recommendations, letters of recommendation, and personal statements gained in importance.

합격률 자료를 보아도, 서로 다른 학업성취도의 학생들이 다양하게 의과대학에 입학하고 있다. 

Acceptance rate data provide additional support for this change, showing that applicants with different levels of academic preparedness (i.e., the combination of cumulative UGPA and MCAT total score) are accepted into medical school. Together, these data and the high importance ratings given to both academic and nonacademic data in the 2008 survey suggest that many medical schools are conducting a more holistic admissions process than they have in the past.

대학마다 학점의 인플레이션이 있다는 것이 보고되고 있다. 그러나 이러한 인플레이션에 대한 정보가 UGPA 활용방식에 영향을 주었는지는 불분명하다.

Additionally, grade inflation occurs at many undergraduate institutions.26 It is unclear whether knowledge of such inflation affects admission officers’ use of UGPAs in their decision making;

Acad Med. 2013 May;88(5):672-81. doi: 10.1097/ACM.0b013e31828bf252.

An overview of the medical school admission process and use of applicant data in decision making: what haschanged since the 1980s?

의과대학 학생선발: 풀리지 않는 과제

Selecting medical students: An unresolved challenge*

DAVID POWIS

The University of Newcastle, Australia

거의 모든 국가에서 의과대학에는 학업적으로 우수한 지원자가 몰림에도 불구하고, 의과대학생 선발위원회가 부적절한 판단을 내렸음을 지적하는 기사나 논문이 종종 등장하여 논란을 불러일으키곤 한다. 가장 우수한 학생을 선발한다면 이들이 좋은 의사가 될 것이라는 가정이 틀린 것일까? 아마도 이제는 초점을 옮겨야 할 때가 왔을지도 모른다. 최고로 우수한 지원자들 사이에서 차이를 내 보려는 기존의 노력 대신, 비학업적 인적특성에 초점을 둬서 부적절한 학생이 입학하지 못하도록 하려는 노력이 필요하다. 의과대학 학생선발의 문제를 분석하고 해결책을 제시하고자 한다. 이 해결책은 이미 70년전에 제시된 바 있지만 다만 도입되지 않았을 뿐인 해결책이다. 이러한 접근법을 도입하지 않는다면 논쟁은 끝나지 않을 것이다. 

Despite the abundant supply of academically outstanding applicants to medical schools in most countries the regularly recurring debate in the academic literature, and indeed sometimes in the popular media, implies that admissions committees are still getting it wrong in a significant number of instances. How can this be so when our procedures are directed unashamedly at selecting the most highly academically and intellectually qualified students in the expectation that they will make the best doctors? Perhaps it is time for a radical change in emphasis. Instead of endeavouring to differentiate among the top ranks of a pool of outstandingly qualified applicants, the selection effort might be better focused on identifying those potentially unsuitable in terms of their non-academic personal qualities to ensure they do not gain entry. The account that follows is an analysis of the problems of medical student selection and offers a potential solution - a solution that was first suggested in the medical literature 70 years ago, but not adopted. It is the present author's contention that the cycle of debate will continue to recur unless such an approach is pursued.

Seventy years ago, in 1944, the UK government-appointed Goodenough Committee of review into medical education reported that: [some medical students] ‘‘who, though able to pass examinations, have not the necessary aptitude, character or staying power for a medical career’’. With the Goodenough Committee report to hand, Smyth, writing in the British Medical Journal in 1946, wrote that ‘‘Existing methods of selection [of medical students] which worked well in the past may no longer be the best possible in changing conditions’’. He added: ‘‘The recent reports . . . have drawn attention to some of the problems connected with the selection of medical students . . . [and] point out the problems, without discussing ways and means of solving them’’ (Smyth 1946).

현재 어떻게 뽑고 있는가?

How were, and are, medical students selected?

The first hurdle to clear to enter medical school in most countries, both in the past and the present, is an academic achievement barrier.

There is a reasonable basis for requiring the academic hurdle: McManus et al. (2013) have described eloquently the predictive link between past academic achievement and academic progress through medical school and beyond. Their ‘‘academic backbone’’ model elegantly demonstrates the link (Figure 1).

Putting the prior academic achievement criterion into perspective a systematic review by Ferguson et al. (2002) showed that academic scores account for 23% of the variance of progress measures at medical school, but only 6% beyond medical school.

이상한 의사들 

Unsatisfactory doctors

Although everyone would freely acknowledge that many doctors are excellent, and most are entirely capable and competent, it is evident that some are not!

• Some doctors have been convicted of criminal acts.

• Some doctors are deficient in communication skills, failing to communicate adequately or appropriately with peers, mentors, patients or patients’ families. 

• Some doctors are unprofessional

• Data from the GMC annual report in 2005 showed that 1 in 15 doctors in the UK were dependent on alcohol or drugs at some stage of their professional lifetime.

• Lastly, the professional competence of some doctors is seriously compromised by mental health issues.

원인으로는 아래와 같은 것들이 있음.

Willcock et al. (2004) suggested that the high incidences of distress and burnout can be attributed inter alia to a doctor’s stressful work environment, their long working hours, possible conflict between work and personal life tasks and their individual psychological vulnerability, a view echoed by Wallace et al. (2009) in their literature review.

이상한 학생들

Unsatisfactory medical students

Most medical educators have had experience of students who cause concern. They comprise a small proportion of any cohort, and may even be progressing adequately academically through medical school, but they exhibit attitudes and/or behaviours that many would consider unacceptable in an aspiring doctor.

몇몇은 유급을 하기도 하지만, 유급의 이유가 늘 학업문제 때문인 것은 아니다.

Notwithstanding the exceptionally high academic standard required to gain entry to medical school it is clear that some students do fail academically during their course. (...) Perhaps the reason for their academic failure lies beyond their academic ability.

자살 충동

And in Australia journalist Amanda Davey, quoting Jessica Dean, President of the Australian Medical Students’ Association (Davey 2014), stated ‘‘that mental illness [is] rife among med students’’ – ‘‘one in five admitting suicidal thoughts in the last year’’ [of their studies].

부적절한/프로페셔널하지 못한 행동

Besides distress to the students concerned ‘‘burnout’’ has been shown to be associated with unprofessional behaviour (Dyrbye et al. 2010), manifested by poor reliability and responsibility, poor initiative and motivation and a severely diminished capacity for self-improvement. Unprofessional behaviour at medical school has been shown to have a strong association with subsequent disciplinary action by a medical board Papadakis et al. (2005).

지원자에 대해서 우리가 알고 있는 것은? 

What do we know about Medical School applicants?

면접에 최종 합격에 중요하나, 전적대학 GPA가 면접 대상자 선정에 가장 중요하다.

In the USA and Canada an overview of medical school admission processes (based on data from 120 informed respondents) by Monroe et al. (2013) reported that although an interview recommendation was the most important determinant (4.5/5) of a candidate receiving an offer, the cumulative undergraduate GPA determined who were invited to interview.

학업적으로 우수한 지원자가 이렇게 많다면, 그들 사이에 구분을 어떻게 할 것인가?

Given the surfeit of academically well-qualified applicants how do medical schools currently differentiate between them?

지금까지는 학업적 수월성에 주로 관심을 두고 있다. 그러나 과연 이것이 옳은가에 대한 의문은 끊이지 않는다.

The focus is clearly still on identifying academic excellence and cognitive (reasoning) skills with only a fairly superficial and subjective investigation of the applicants’ personal qualities. The quest for the presumed ‘‘best’’ applicants remains the prime focus. Contemporary commentators have continued to ask the question: Is this the right way? (Hughes 2002; Powis 2003, 2008; James et al. 2010; Mercer & Puddey 2011; Wilson et al. 2012; Eskander et al. 2013; Leinster 2013).

Barr (2010), writing in The Lancet, ‘‘found no scientific evidence that supported the power of performance in undergraduate science courses as a way to predict clinical or professional quality as a physician’’ and ‘‘found . . . consistent evidence that performance in the premedical sciences is inversely associated with many of the personal, non-cognitive qualities so central to the art of medicine’’.

패러다임 시프트가 필요한가? 

Time for a paradigm shift?

The vast majority of medical school applicants are more than adequately equipped academically for their studies and their later professional careers.

기존의 선발도구로 부적합한 학생을 가려낼 수 있겠는가?

The practical question now becomes: Can we identify the potentially unsuitable at the outset using tools currently or

potentially available? 

• academic record
• cognitive skills tests
• personal statement
• referees’ reports
• interview – panel, MMI
• non-cognitive tests (personality measures)

학업 기록을 통해서 의과대학 학업을 따라가기에 어려워보이는 지원자를 가려낼 수 있다.

The academic record will identify those potentially academically inadequate to undertake medical studies.

일정 수준 이상의 학업성취도를 보유한 지원자를 가려냈다면, 이후에는 이에 따른 영향이 없어야 한다. 그렇게 된다면 사회경제적 수준이 낮은 인구집단의 지원자에게도 좀더 입학기회가 확대될 것이다. 

It could be argued that academic marks above the threshold should play no further part in student selection. This would have positive effects on widening access to applicants from lower socio-economic groups (Powis et al. 2007).

인지적 역량 Cognitive skills

보통 직원을 선발할 때는 지능이 직무수행능력의 가장 좋은 예측인자라고 하지만, 의과대학에서는 인지검사가 이후 수행능력 예측이라는 측면에서 선발에 별 도움이 되지 않는다. 

In the general field of employment selection, it has long been recognised that ‘‘Intelligence is the best predictor of job performance’’ (Ree & Earles, 1992). However, the cognitive skills tests used in medical school selection procedures have usually been found to add little to academic scores in predicting outcomes

반면, GPA에 추가적으로 MCAT점수를 보는 것은 도움이 된다.

On the other hand, in the opinion of the respondents to the survey of Monroe et al. (2013), MCAT does offer added value to GPA scores.

자기소개서 Personal statements

여러가지 문제가 있음.

There are many problems that undermine the usefulness of personal statements for informing medical school selectors.

추천서 Referees’ reports

신뢰도는 높일 수 있으나 타당도가 낮다.

Munro et al. (2012) have asserted that referees’ reports have low validity even when structured to increase reliability.

개별 추천인들의 내적일관성은 높지만, 추천인간 비교에 있어서 일치도가 매우 낮다.

The study found that while the internal consistency of individual referees responses was high, agreement between the referees in terms of individual candidates’ strengths and weaknesses was very low, almost as if they were evaluating different people.

그러나 낮은 점수에 있어서는 일치도가 충분한 것들이 있어서, 부족한 지원자를 걸러내는데는 좋은 방법이 될 수 있다.

Interestingly, there was sufficient agreement on the small number of low scores to suggest that such referees’ reports may be better suited for deselecting weak candidates.

면접 Interviews

어떻게 구조화되느냐에 따라서 신뢰도가 다르다.

An interview is a commonly included component of medical student selection procedures. However their reliability can be low depending, in part, how they are structured.

패널 인터뷰는 MMI에 비해서 신뢰도가 낮다.

Panel interviews have been shown to be rather less reliable than the ‘‘multiple mini-interview’’ (MMI) procedure, first described in the medical student selection context in articles from McMaster University, Canada (see Eva et al. 2004, and subsequently).

비인지역량 검사 Non-cognitive tests

The Newcastle, Australia medical school has focused on the following qualities to develop a battery of tests that has become known as the Personal Qualities Assessment (PQA, www.pqa.net.au):

• Moral orientation (on a continuum ranging from ‘‘libertarian’’, the rights and needs of individuals versus ‘‘communitarian’’, the expectations and needs of society as a whole).
• Resilience (vs. inability to cope with stress; emotionally volatile, ‘‘neurotic’’).
• Self-control (conscientious, orderly, restrained, industrious vs. disorderly, unrestrained, unreliable, impulsive, permissive, anti-social).
• Involvement (empathic, confident in dealing with others, co-operative, agreeable vs. aloof, narcissistic, disagreeable, manipulative, uncomfortable with others).

PQA 점수분포에서 상 하위 2.5% 안에 들어가는 학생은 선발에서 배제될 것을 고려해볼 수 있다.

The working hypothesis is that those individuals represented in the extreme region of the trait score distribution, the top and bottom 2.5%, should be excluded from consideration for entering medical school.

A model for medical school selection

In conclusion I present a model for medical student selection proposed by Bore et al. (2009), a model which Smyth (1946) would surely have approved as evidenced by his remark:

‘‘We want . . . two independent tests, or sets of tests – the one for ability, the other for character’’.

• Besides Selecting in for
• academic ability (academic record)
• cognitive skills (‘‘aptitude’’ tests)
• ability to communicate appropriately (interview)
• good interpersonal skills (interview)

• Select out (non-cognitive tests) those applicants who
• demonstrate traits of psychological vulnerability (inability to handle stress appropriately; low resilience)
• high levels of neuroticism
• low levels of conscientiousness
• extreme detachment, extreme emotional involvement
• high levels of impulsiveness and permissiveness

Med Teach. 2015 Mar;37(3):252-60. doi: 10.3109/0142159X.2014.993600. Epub 2014 Dec 23.

Selecting medical students: An unresolved challenge.

의과대학 학생선발에서 면접관의 편향

Interviewer bias in medical student selection

Barbara N Griffin and Ian G Wilson