의과대학 입학생의 성공에 필요한 핵심 인성역량 : 무엇이고, 어떻게 입학단계에서 평가할 수 있는가?(Acad Med, 2013)

의과대학 입학생의 성공에 필요한 핵심 인성역량 : 무엇이고, 어떻게 입학단계에서 평가할 수 있는가?(Acad Med, 2013)

Core Personal Competencies Important to Entering Students’ Success in Medical School: What Are They and How Could They Be Assessed Early in the Admission Process?

Thomas W. Koenig, MD, Samuel K. Parrish, MD, Carol A. Terregino, MD, Joy P. Williams, Dana M. Dunleavy, PhD, and Joseph M. Volsch, MPA

입학시에 의과대학 입학생이 지녀야 할 학업적 역량에 대해서는 일반적 합의가 있다.

There is general agreement in the medical education community about the academic competencies that medical students should demonstrate when they matriculate.

비록 의학교육계는 졸업시에 갖춰야 하는 인성역량에 대래서는 동의를 이뤘으나, 입학시에 어떠해야 하는가에 대해서는 합의가 없다.

Although the community has agreed on the personal competencies that medical students should demonstrate at graduation,1 it has not reached consensus on those that are important at entry or how to incorporate them into the admission process.

Albanese 등은 입학과정에서 87개 이상의 서로 다른 인성특성이 평가되고 있다고 추정하였다. 의과대학간 합의가 이토록 부족한 것은 어떤 인성역량이 의과대학 성과와 정의 관계게 있다는 연구 결과를 감안하면 무척 놀라운 것이다.

Albanese and colleagues2 estimated that more than 87 different personal qualities are assessed during the admission process. This lack of consensus among schools is surprising given that research has linked certain personal competencies to positive admission and medical school outcomes.

예컨대 Carrothers 등은 다음의 인성이 의과대학에서 주로 평가된다고 했다.

For example, Carrothers and colleagues3 found that 

• having good interpersonal skills, 
• knowing one’s emotions, 
• recognizing emotions in others, 
• possessing the ability to manage one’s emotions in difficult situations, and 
• being able to motivate oneself 

were frequently cited by medical school admission committee members as desirable attributes for prospective medical students. 

비슷하게 Adams 등은 의과대학 교수/전공의/학생들은 의과대학에서의 성공에 중요한 요소로 다음의 것을 꼽았다고 했다.

Similarly, Adams and colleagues4 found that demonstrating 

• motivation, 
• a desire to learn, 
• integrity and ethics, 
• self-management, and 
• strong interpersonal and 
• teamwork skills 

were reported by medical school faculty members, residents, and students as being important to success in medical school.

연구 결과를 살펴보면 어떤 인성 특징든 환자 성과, 환자들의 의사에 대한 평가에도 연관된다고 했다.

Researchers have related some of these personal characteristics and skills to improved patient care outcomes and to patients’ ratings of their physicians.5,6

관련된 연구 결과로는 다음과 같은 것이 있다.

• For example, good teamwork and collaboration are correlated with improved patient outcomes, patient satisfaction, and greater job satisfaction among physicians.7 
• Patients who report being treated with dignity by their physicians are more likely to adhere to treatment plans and to be satisfied with their care.8 
• Similarly, physicians who “adopt a warm, friendly, and reassuring manner” with their patients are more effective than those who keep consultations formal and do not offer reassurances.9 
• Recently, Hojat and colleagues10 found that patients of physicians with high levels of empathy have better health outcomes than patients of physicians with moderate and low levels of empathy. 
• Moreover, when physicians’ personal skills are lacking, negative professional outcomes are likely. 
• For instance, Papadakis and colleagues11 showed that unprofessional behavior in medical school (e.g., irresponsibility, lack of capacity for self-improvement) predicts later disciplinary action by state medical boards.

의과대학 입학에서 인성역량의 역할

The Role of Personal Competencies in Medical School Admissions

AAMC의 리더들과 다른 의학교육계 사람들은 지원자의 인성 특성을 더 강조해야 한다고 말했다.

Leaders of the Association of American Medical Colleges (AAMC)12 and others in the medical education community have called for more emphasis to be placed on applicants’ personal competencies in the admission process.

자료를 보면 면접까지 가기 전에 상당한 수의 지원자가 제외된다. 2011년, 한 지원자는 평균 14개의 의과대학에 지원했으나 평균적으로 2개 이하의 학교에서만 면접까지 갔다.

Data show that a significant part of admission screening takes place before interviews: In 2011, the average applicant submitted 14 applications but received less than 2 interview invitations.13

이 문제를 해결하기 위해서 가장 먼저 해야 할 것은 평가할 인성역량을 정하고, 의학교육계의 수요와 목적의 균형에 맞는 실무적, psychometric적 이슈를 해결하기 위한 도구에 합의해야 한다.

To meet this challenge, the medical education community must first agree on a universal set of personal competencies to measure as well as a set of tools that balances the needs and goals of the admission community with practical (e.g., cost, accessibility) and psychometric issues.

Defining Core Personal Competencies for Entering Medical Students

비록 역량을 정의하고자 했던 시도는 없었던 바 아니나(AAMC는 1970년대와 1990년대에 시도한 바 있음), 의학교육계 내에서 이에 대한 합의를 이루고자 하는 시도는 거의 없었다. 따라서 AAMC는 다년간의 상당한 노력을 들여서 핵심 인성역량을 규명했다.

Although there have been attempts to systematically define the personal competencies that medical school matriculants should demonstrate on entry (e.g., the AAMC explored this in the 1970s and 1990s),16 there has been little effort to build consensus about these competencies in the wider medicaleducation community. Therefore, the AAMC undertook a rigorous, multi- year process to research and identify core personal competencies for students entering medical school in the 21st century. This process (Table 1) 

16 Etienne PMJ, Julian ER. Assessing the personal characteristics of premedical students. In: Camara WKE, ed. Choosing Students: Higher Education Admissions Tools for the 21st Century. Mahwah, NJ: Lawrence Erlbaum Associates; 2005.

의과대학 성공에 중요한 인적 특성

Identifying personal characteristics important to success in medical school

MR5 Committee는 두 개의 설문 진행

The MR5 Committee began the process by conducting two surveys

• In 2008, U.S. and Canadian admission officers were asked to describe their school’s admission process and to rate the importance of 41 personal characteristics to success in medical school.
• In 2009, U.S. and Canadian academic affairs officers were asked to rate the importance of 72 characteristics to success in medical school. Data from these two surveys17,18

핵심 인성역량 개발

Developing the set of core personal competencies

다음으로, ILWG는 직무분석을 했다.

Next, the ILWG conducted a multistep job analysis to identify the core set of personal competencies

각각의 인성특성에 대해서 다음의 질문을 던졌다.

We then asked the following questions about each personal characteristic: 

• 1. Is this characteristic related to medical student performance, particularly the behaviors associated with success in medical school? 
• 2. Do students need to display this characteristic at entry into medical school? 
• 3. Is it reasonable to assume that medical school applicants can demonstrate this characteristic? (Is it developmentally appropriate?) 
• 4. Is this characteristic fixed, or is it malleable? Is it something that medical education can build on as the student matures and is exposed to new experiences?

이 질문에 대한 답변에 따라 하위집단을 선택함

On the basis of the answers to these questions, we selected a subset of personal characteristics to develop into core personal competencies.

평균적으로 매우-극도로 중요하다고 평가하였으나, 입학과정에서 관련되어 제공되는 정보의 질에는 만족하지 못함

As shown in Table 2, on average, all of the draft personal competencies were rated by admission officials as “very important” to “extremely important.” Respondents were not, however, satisfied with the quality of information available about these competencies during the admission process.

피드백 수집

Collecting feedback on the core personal competencies

The ILWG’s recommendation served as the foundation for the AAMC Admissions Initiative. One of that group’s first projects was to review the ILWG’s draft definitions of the recommended core personal competencies.

9개의 핵심인성역량 승인

Approving the nine core personal competencies for entering medical students

2013년 2월 AAMC COA는 최종 9개 리스트를 만들었다. 

In February 2013, the AAMC’s COA endorsed the final list of nine core personal competencies for entering medical students (defined in Table 4): 

• ethical responsibility to self and others; 
• reliability and dependability; 
• service orientation; 
• social skills; 
• capacity for improvement; 
• resilience and adaptability; 
• cultural competence; 
• oral communication; and 
• teamwork.

평가도구 검색

Exploring Tools to Assess the Core Personal Competencies Early in the Admission Process

비록 ILWG 설문이 인성역량을 평가하는 도구도 제안하긴 했지만, 그것의 활용과 가치에 대해서는 아직 답해야 할 질문이 많다.

Although the ILWG survey suggested a desire among admission officers for tools that assess applicants’ personal competencies early in the admission process, there are many unanswered questions about the use and value of such measures in medical school admissions.22

50개 이상의 논문, 6개의 미발표된 보고서 등등을 보았음.

We identified more than 50 seminal articles (including several meta-analyses) and six nonpublished technical reports about tools currently used to measure personal competencies in higher education and employment settings. We made subjective, holistic judgments about tools’ potential to provide information on applicants’ core personal competencies for use in the pre-interview screening stage of the admission process. 

여섯 개의 도구를 여덟 개의 준거로 평가하였다.

We judged six types of tools according to the following eight criteria: 

• validity, 
• reliability, 
• group differences, 
• susceptibility to faking and coaching, 
• applicant reactions, 
• user reactions, 
• cost/resource utilization, and 
• scalability for use in pre-interview screening (Appendix 1).

SJT

Situational judgment tests

SJT의 진행방식. 평가자는 답을 골라야 하고 포멧이 다양할 수 있다.

In situational judgment tests (SJTs), examinees are asked to indicate how they would (or should) respond to dilemmas presented in text-based, video, or animated scenarios. Response formats vary: 

• Examinees may be asked to select from multiple-choice options, 
• identify the most and least effective responses, 
• and/or answer open-ended questions. 

캐나다, 벨기에, 이스라엘 등에서 활용되고 있음

SJTs have been used in medical school admission processes in Canada (the CASPer assessment23), Belgium,24–26 and Israel.27

직원 채용에 관한 문헌을 보면 SJT의 신뢰도 타당도 근거가 충분하다. 1997년부터 의과대학 선발에 사용해온 벨기에에서의 결과도 마찬가지이다. 추가적으로 영국에서의 연구결과를 보면 SJT가 의사로서의 수행역량을 예측하며, 임상에서의 문제해결 시험에 incremental validity를 제공한다. 또한 지원자도 SJT에 호의적이다.

The employment literature28 provides strong evidence for the reliability and validity of SJTs, as does research conducted in Belgium,26 where an SJT has been used in the medical school admission process since 1997. Additionally, research from the United Kingdom shows that SJT scores predict competency-based ratings of physician performance and provide incremental validity above and beyond a clinical problem-solving test.29 Further, applicants hold generally positive attitudes about SJTs.30

SJT수행에 약간의 인종/민족간 차이가 있을 수 있다. 그러나 법과대학 결과를 보면 AA나 Latino의 입학을 더 높여준다.

There is some evidence suggesting that there may be small racial/ethnic group differences in performance on SJTs that emphasize decision making.31 However, research conducted by the College Board and the Law School Admission Council indicates that including these tests in the admission process may increase the percentage of African American and Latino matriculants compared with using academic data alone, and that performance on SJTs is the best predictor of “lawyering effectiveness.”32,33

SJT는 개발에 비용이 많이 들며, 시나리오 개발에 기술전문가가 필요하다. 그러나 면접 전 스크리닝에 scalable하며 대규모의 지원자에게 시행할 수 있다. 결과 자료도 활용하기 용이하다.

SJTs are somewhat expensive to develop because of the technical expertise needed to create and score scenarios. However, they are scalable for use in pre-interview screening because they can be administered to a large number of applicants before the interview. Further, when SJTs are scored, data are presented in a format that is easy to consume.

표준화 수행 평가(SEP)

Standardized evaluations of performance

평가자는 그래픽, 상호비교, 행동-관련-평가스케일을 활용하여 지원자를 특정 역량에 따라 평가한다.

In standardized evaluations of performance (SEPs), raters use a graphic, comparative, or behaviorally anchored rating scale to evaluate applicants on a set of competencies.

대부분의 의과대학 입학절차에서 비표준화 추천서를 사용하지만, 비학업적 변인에 대해서는 평가자간 신뢰도가 낮고, 예측 타당도도 낮고, comparative data도 부족하다. 다른 전문직 교육과정에서는 SEP을 사용해왔으며 ETS에서는 PPI를 도입했다. 

Although most medical school admission processes use nonstandardized letters of recommendation—which have poor interrater reliability for nonacademic variables,34 have poor predictive validity, and lack comparative data—other graduate and professional program (e.g., veterinary medicine, optometry, physical therapy) admission processes use SEPs. In 2009, the Educational Testing Service introduced the Personal Potential Index,35 an SEP for use in graduate admissions, but there is no published literature to date on its psychometric properties.

의과대학 지원자의 수행능력을 보면 SEP 점수와 작지만 유의미한 정의 상관관계를 확인할 수 있다. 입학담당자는 SEP에 긍정적인 태도를 가질 가능성이 높은데, 왜냐하면 평가자들이 지원자의 인성역량을 묘사할 구체적인 행동 예시를 포함시켜야 하기 때문이다.

Research on the Medical Student Performance Evaluation shows small but significant observed positive correlations between standardized evaluations and performance on comprehensive clinical performance exams.36 Admission officers are likely to have positive attitudes about SEPs because raters must include specific examples of behaviors illustrating applicants’ personal competencies.37

점수를 잘 주는 평가자가 있을 수 있고 평가에 variance가 있을 수 있다.

There is potential for rater leniency and consequent lack of variance in ratings, though.

성취기록

Accomplishment records

성취기록은 자기소개서와 비슷하며, 성취와 경험의 표준화된 기술이다. 지원자는 중요한 인성역량과 관련한 행동을 기술하도록 한다.

Accomplishment records, also known as autobiographical questionnaires, are standardized descriptions of achievements and experiences. Applicants are asked to describe behaviors related to a set of important personal competencies.

신뢰도는 이러한 성취기록이 감독관이 있는 상황에서 수집되고, 다수의 평가자가 평가하여 확보할 수 있다. 타당도 자료는 수집 불가능하다. 지원자와 의과대학은 미적지근할 수 있는데 일이 늘어나기 때문이다. 점수를 부여하지 않은 성취기록에 대해서는 거의 문헌자료가 없으나 개발이 어렵지 않고 다수의 지원자에게 적용가능하다.

Reliability is best when accomplishment records are collected in proctored settings and are scored by multiple raters.39 Validity data are not available with respect to their use in admissions. Applicants and users may have lukewarm reactions to them because of the added workload. There is little published research on unscored accomplishment records, but they are inexpensive to develop and can be administered to large numbers of applicants.

인성검사, 전기적 자료

Personality and biographical data inventories

인성검사도구와 전기적 자료는 지원자에게 리커트 스케일로 특정 명제에 대해서 어느 정도로 자신이 해당되는지 평가하게 한다. 이들 도구는 상대적으로 저렴하다.

Personality inventories and biographical data inventories ask applicants to indicate the extent to which a series of statements accurately describe them, typically using a Likert-type response scale. These tools are relatively inexpensive to develop and can be administered to large numbers of applicants.

그러나 high-stake 맥락에서 우려가 있다. 가장 큰 것은 코칭과 faking이다.

However, there are concerns about their use in a high-stakes admission context. A primary concern is the potential for coaching and faking responses.

낮은 SES의 지원자는 이러한 코칭을 받지 못해 불이익이 있을 수 있다.

Applicants from low socioeconomic backgrounds who do not have access to such coaching may be at a disadvantage.

면접

Local interviews

인터뷰 종류는 비구조화에서 구조화까지 다양하나 대부분 반구조화 면접을 사용한다.

Interview types range from unstructured to structured, but most medical school interviews are semistructured.

면접은 한계가 많은데, 우선 비구조화 면접의 신뢰도가 낮으며, 지원자의 지원서 자료를 같이 면접관에게 제공하는 평가에 오류가 생길 가능성이 높다. 또한 평가자 오류가 있을 수 있고, 지원자보다 평가자에 따라 점수가 달라진다.

Local interviews have a number of limitations, however. Reliability for unstructured interviews is poor, and the practice of providing interviewers with access to applicants’ application data introduces bias.42,43 In addition, local interviews are subject to rater error, and ratings may have more to do with the interviewer than the interviewee.43 

구조화된 면접은 다음과 같은 것이 있다.

structured interviews conducted at the University of Iowa Carver College of Medicine,47 and “behavioral event interviews” used by the Scholarly Excellence, Leadership Experiences, Collaborative Training program at the Morsani College of Medicine48

마찬가지로 코칭과 faking에 대한 우려

One concern about interviews is the potential for coaching and faking.

AC

Assessment centers

AC는 다양한 표준화된 평가를 활용할 수 있다.

Assessment centers can employ several standardized exercises (e.g., interviews, role-plays, in-baskets, group discussions) to provide multiple opportunities for multiple raters to evaluate applicant behaviors.

다양한 자료가 수집가능하나 자원이 많이 든다.

Data from assessment centers provide important information about applicants’ personal competencies, but such centers are resource intensive. Thus, it is not feasible to conduct them on a national level to provide data in time for pre-interview screening.

추가 연구 권고사항

Tools recommended for future study

우리의 권고는 SJT, SEP, 성취기록이다.

After reviewing the literature and evaluating potential tools on the eight criteria, we suggested that the AAMC further investigate three tools for possible use in assessing applicants’ core personal competencies during the admission process: SJTs, SEPs, and accomplishment records. We recommended these tools because each of them

그 이유는 아래와 같다.

• provides data about personal competencies in a format that is easy to use and would be available in time for pre-interview screening, 

• allows for multiple sources of assessment, 

• has acceptable validity and is likely to provide predictive value beyond UGPAs and MCAT scores in predicting nonacademic outcomes, 

• demonstrates less potential risk of coaching and faking effects compared with other tools, 

• is likely to be accepted by applicants and admission officers, and 

• avoids exorbitant costs that would likely be passed on to applicants.

어떤 도구도 모든 상황에 완벽하지 않아서 다양한 도구를 사용할 것을 권고한다. SJT SEP 성취기록은 모두 toolbox로서 함께 사용되어야 한다.

No tool is perfect for all situations, so we recommend that multiple tools be employed to assess personal competencies to enable admission officers to evaluate the information collected (just as they currently consider both UGPAs and MCAT scores in context). SJTs, SEPs, and accomplishment records should be used together—as part of an “admissions toolbox”—along with data on applicants’ academic competencies, in deciding which applicants to interview.

SJT에 대한 연구가 더 필요하다. 

Future research on the use of SJTs in medical school admissions should explore different formats for 

• presenting scenarios (e.g., actors, avatars), 
• alternative response formats (e.g., rank order, narrative responses), 
• validity, and 
• the impact of coaching/faking on validity and user acceptance.

2 Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78:313–321.

22 Bardes CL, Best PC, Kremer SJ, Dienstag JL. Perspective: Medical school admissions and noncognitive testing: Some open questions. Acad Med. 2009;84:1360–1363. Acad Med. 2009;84:1360–1363. 

SJT 23 Dore KL, Reiter HI, Eva KW, et al. Extending the interview to all medical school candidates—Computer-Based Multiple Sample Evaluation of Noncognitive Skills (CMSENS). Acad Med. 2009;84(10 suppl):S9–S12. 

SJT 24 Lievens F, Sackett PR, Buyse T. The effects of response instructions on situational judgment test performance and validity in a high-stakes context. J Appl Psychol. 2009;94:1095–1101. 

SJT 25 Lievens F, Sackett PR. Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms. J Appl Psychol. 2007;92:1043–1055. 

SJT 26 Lievens F, Sackett PR. The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. J Appl Psychol. 2012;97:460–468. 

SJT 27 Ziv A, Rubin O, Moshinsky A, et al. MOR: A simulation-based assessment centre for evaluating the personal and interpersonal qualities of medical school candidates. Med Educ. 2008;42:991–998.

SEP 35 ETS. ETS Personal Potential Index. http:// www.ets.org/ppi. Accessed January 25, 2013.

Structured Interview 48 Carney A. Building a better doctor. USF Magazine. 2011:53(3). http://www.magazine. usf.edu/2011-fall/building-a-better-doctor. aspx. Accessed January 14, 2013.

---

Acad Med. 2013 May;88(5):603-13. doi: 10.1097/ACM.0b013e31828b3389.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

Koenig TW1, Parrish SK, Terregino CA, Williams JP, Dunleavy DM, Volsch JM.

Author information

• 1Johns Hopkins University School of Medicine, Baltimore, Maryland, USA.

Abstract

Assessing applicants' personal competencies in the admission process has proven difficult because there is not an agreed-on set of personalcompetencies for entering medical students. In addition, there are questions about the measurement properties and costs of currently available assessment tools. The Association of American Medical College's Innovation Lab Working Group (ILWG) and Admissions Initiative therefore engaged in a multistep, multiyear process to identify personal competencies important to entering students' success in medical school as well as ways to measure them early in the admission process. To identify core personal competencies, they conducted literature reviews, surveyed U.S and Canadian medical school admission officers, and solicited input from the admission community. To identify tools with the potential to provide data in time for pre-interview screening, they reviewed the higher education and employment literature and evaluated tools' psychometric properties, group differences, risk of coaching/faking, likely applicant and admission officer reactions, costs, and scalability. This process resulted in a list of ninecore personal competencies rated by stakeholders as very or extremely important for entering medical students: ethical responsibility to self and others; reliability and dependability; service orientation; social skills; capacity for improvement; resilience and adaptability; cultural competence; oral communication; and teamwork. The ILWG's research suggests that some tools hold promise for assessing personal competencies, but the authors caution that none are perfect for all situations. They recommend that multiple tools be used to evaluate information about applicants' personalcompetencies in deciding whom to interview.

PMID:

 

23524928

 

[PubMed - indexed for MEDLINE]