의과대학 학생선발에서의 인적특성(personal qualities, personal characteristics) 평가
Assessing Personal Qualities in Medical School Admissions
Mark A. Albanese, PhD, Mikel H. Snow, PhD, Susan E. Skochelak, MD, MPH, Kathryn N. Huggett, MA, and Philip M. Farrell, MD, PhD
의과대학 학생으로 누구를 선발할 것인가를 결정할 때에는 철저한 검토가 반드시 필요하다. Jordan Cohen은 학부 GPA와 MCAT점수를 학생 선발의 일차적 수단으로 활용하는 것을 비판했다. 그 대신 "MCAT와 GPA는 일정 수준 이상인지만 보면 된다"라고 주장하며 "학생선발절차는 일차적으로 인적특성을 보는 것으로 시작하고, GPA와 MCAT점수는 그 다음 평가에서 봐야 한다"라고 말했다. 그는 또한 이러한 접근법에 대해서 "입학위원회는 한두개의 학업적 흠(blemish)를 우수한 인적특성으로 극복한 사례를 무수히 봐왔을 것이다"라고 말했다.
Making decisions about whom to admit to medical school has come under increasing scrutiny. Jordan Cohen, MD, in his address to the 112th annual meeting of the Association of American Medical Colleges, decried the use of undergraduate grade-point averages (GPAs) and Medical College Admission Test (MCAT) scores as the primary means of selecting medical students. Instead of using these indicators, he suggested using “MCAT scores and GPAs only as threshold measures,” or “beginning the screening with an assessment of personal characteristics and leave the GPAs and MCAT scores 'til later.” He argued that using this approach “admission committees might well find many instances in which truly compelling personal characteristics would trump one or two isolated blemishes in the academic record.”1
Cohen의 주장을 받아들이기 위해서는 인적특성을 효과적으로 측정하는 것이 더더욱 중요해진다. 인적특성 평가를 위한 의과대학 면접에 관한 문헌과 다른 인적특성 평가접근법을 살펴봄으로써 이 이슈에 대한 접근법을 제안하고자 한다.
With Dr. Cohen's challenge to look first to personal qualities in the admission process, the need to effectively measure personal qualities has assumed greater importance. In this article, we review the literature on the medical school interview as a mechanism for assessing personal qualities, discuss the challenges in using the interview and other approaches to assessing personal qualities, and then suggest approaches that might be taken to address this important issue.
BACKGROUND
전통적으로 인터뷰는 인적특성을 평가하기 위한 주된 수단이었으며, 의과대학 입학면접은 일부 소수 의과대학을 제외하고 거의 모든 의과대학에서 활용되어왔다. Edwards등은 의과대학 면접의 네 가지 목적에 대하여 기술한 바 있다. - 정보수집, 의사결정, 확인, 모집 -. 그 중 가장 중요한 목적은 다른 수단으로는 수집하기 어려운 지원자의 비학업적 정보를 얻는 것이다. 그들이 주장한 정보수집방법은 SAMS라는 것으로, 직무분석을 바탕으로 면접 내용을 정하는 방식이다. 이에 따라 모든 지원자에게 활용할 문항을 표준화하고, 평가자에게는 질문에 대한 예시응답을 제공하여 일관된 평가를 할 수 있게 해준다. 그리고 면접은 일군의 평가자에 의해서 이루어진다. 면접은 '구조화', '반구조화', '비구조화' 면접으로 분류할 수 있다.
Historically, the interview has been one of the primary methods of assessing personal qualities. Interviews for admission to medical school are conducted by all but a few U.S. medical schools.2,3 Edwards et al.3 cite four purposes for the admission interview: information gathering, decision making, verification, and recruitment. They argue that the most important purpose of the interview is to gather non-academic information about candidates that would be difficult or impossible to obtain by other means. The method they advocate for obtaining this information is a Success Analysis of Medical Students (SAMS), which includes selecting interview content based upon a job analysis (the critical-incidents technique is advocated for this purpose), standardizing the questions asked of all applicants, providing interviewers with sample answers to questions to help them give consistent ratings, and conducting each interview with a board or panel of interviewers. Interviews have been classified as being “structured” (like the SAMS model), semi-structured (having some but not all elements of a structured model), and “unstructured.”
입학위원회가 면접을 점차 강조하고 있다는 근거는 많다. Purvear와 Lewis는 107개 의과대학 중 61%에서 입학면접이 선발에서 가장 중요한 변수라고 응답한 바 있다고 밝혔다. 경험적 근거 역시 이를 지지하는데, Nowacek 등은 입학위원들은 지원자를 면접한 후에 지원자에 대한 평가를 바꾸는데, 그 변화의 폭이 SD의 0.47배에 달한다고 밝혔다. Patrick 등은 SAMS 모델을 입학결정에 도입한 효과를 발표한 바 있는데, 면접 평가 결과를 포함시키자 합격결정의 variance의 percentage가 21%에서 37%로 증가하였다. 이렇듯 입학면접에서 획득한 정보는 분명 입학결정에 큰 영향을 준다고 볼 수 있다. 그렇다면 어떤 인적특성을 평가해야 하며 어떻게 평가해야 하는 것일까?
Substantial evidence exists that admission committees place great emphasis on the information gleaned from interviews. Puryear and Lewis2 reported that 61% of 107 medical schools responding to their survey stated that the admission interview data were the most important variables used in selection. Empirical data support this result as well. Nowacek et al.4 found that, after interviewing candidates, admission committee members changed their ratings of the candidates, with mean values for various assessed qualities changing by as much as .47 of a standard deviation (effect size or ES). Patrick et al.5 reported the impact of introducing interview data obtained using the SAMS model on admission decisions. After adding the interview ratings to information from the written application, the percentage of variance in acceptance decisions accounted for by the regression model increased from 21% to 37%. Data obtained from the admission interview clearly can have a significant effect on admission decisions, but what are the non-academic qualities being assessed in the interview and in what ways are they being assessed?
여러 연구자가 이에 대해 보고한 바 있다.
- Meridith et al.6 reported rating an applicant's maturity, individual achievement, motivation/interest in medicine, ability, and interpersonal skills.
- Nowacek et al.4 evaluated communication and interpersonal skills, commitment to serve others, familiarity with issues in medicine, leadership ability, motivation for medicine, and overall impression.
- Murden et al.7 assessed applicants' levels of maturity, nonacademic achievement, motivation, and rapport.
- Powis et al.8 assessed perseverance, tolerance of ambiguity, supportive and encouraging behavior, motivation to become a doctor, self-confidence, compatibility with the school's study styles, and an overall judgment.
- Taylor9 reported drawing traits assessed in a written form from 87 positive qualities of successful physicians.
- Collins et al.10 assessed communication, maturity, caring qualities/friendliness, awareness of community, political, social and medical issues, certainty of career choice, involvement in school activities, and involvement in community activities.
- Shaw et al.11 assessed 20 “non-cognitive, non-teachable traits,” such as being honest, energetic, confidence-inspiring, and conscientious. These authors are not alone in their beliefs that certain noncognitive traits are non-teachable.
그러나 Shaw 등만이 '교육불가능한 비인지적 특성'에 대한 믿음을 가지고 있었던 것은 아니다. Bullimore 등은 인성은 18세까지 결정되기 때문에 의과대학 입학면접에서 비인지적특성에 대한 평가를 하는 것이 중요하다고 주장한다. '인성'과 '교육불가능한 특성'이라는 개념은 시간과 상황에 무관하게 일관된 특성이 있음을 뜻한다. 그런데 '정직성'이 정말로 교육불가능한 것인가? 에너제틱한 사람은 늘 그러한가? 어떤 특성이 '교육불가능하다'는 것을 인정하더라도, 그것을 한두시간 면접 내에 보여주는 것이 가능할까?
- Bullimore12 argues that personality is set by age 18, making assessment of noncognitive variables in the medical school admission interview critical. The concept of personality and non-teachable traits implies traits that are stable across time and situations. Is honesty really non-teachable? Is an energetic person always energetic? Even if one accepts that there are some non-teachable traits, might they not be coachable for display in a one- to two-hour interview?
의과대학입학은 대부분의 지원자들에게 '고부담'의 일임에 분명하다. 많은 지원자들이 엄청난 양의 돈을 "MCAT대비 서비스"에 쏟아붓는다. 표준화된 면접을 통해서 교육불가능한 인적특성을 평가하는 것에 대비해서도 유사한 서비스가 가능한 것은 아닐까? 또한 면접이 비인지적 특성을 평가하기 위한 최선의 방법인 것은 맞을까? Taylor는 이런 특성은 지원단계에서 개인이 선택하여 작성할 수 있는 서식을 주고 이를 작성하게 함으로써 평가할 수 있다고 주장한다. 더 나아가 그는 UICM에서 이런 방식으로 선발한 학생이 전통적 면접으로 선발한 학생과 다르지 않음을 보여주었다. UWMS의 지원 단계에서 자기소개서는 비인지적 특성을 보여주는 핵심 표지자 중 하나이다. 그러나 우리의 문헌고찰 결과 면접에 비해서 자기소개서에 대한 평가가 비인지적 특성 평가에 어떤 차이가 있는가를 보여준 연구는 없다.
Admission to medical school is a high-stakes proposition for almost all applicants. Many applicants spend significant sums of money for test preparation services for the MCAT. Might a standardized interview purporting to assess non-teachable skills find itself susceptible to coaching from such a service? Further, is an interview the only way or even the best way to assess these noncognitive traits? Taylor9 argues that such traits can be assessed by having candidates distribute evaluation forms to individuals of their choice as part of the application process. He further reports that the students selected at the University of Iowa College of Medicine using such an approach did not differ from those selected when an traditional interview was conducted. In the application process at the University of Wisconsin Medical School, the personal statement has served as a key indicator of noncognitive traits. Our literature review, however, found no study that examined to what extent admission committees' assessments of the personal statement yielded different assessments of applicants' noncognitive qualities than an interview.
궁극적으로 면접에 비용과 시간을 투자할 가치가 있는가에 대한 결정은 면접이 다른 선발도구를 통해서는 얻을 수 없는 무언가(특히 지원서를 통해서 알 수 있는 정보)를 제공하는가에 달려있다고 할 수 있다. 면접의 가치에 대한 근거로 면접의 신뢰도와 타당도에 대한 연구가 있다. 그러나 그 결과는 모호하다. 면접관의 신뢰도에 대한 연구는 다양한 결과를 보여주고 있다.
Ultimately, the decision whether an interview is worth the time and expense must be based on whether the interview yields something that cannot be obtained by other means, and, in particular, something that cannot be obtained from a review of written application materials. Evidence for the value of the interview has been sought in studies assessing the reliability and validity of the interview. The results have been equivocal. Studies of the reliability of interviewers have produced quite variable estimates.
- Meridith et al.6 found inter-rater correlations ranging from .55 to .91 for five qualities assessed in a sample of 14 applicants, each evaluated by two raters.
- Powis et al.8 report kappa (chance corrected inter-rater agreement) statistics ranging from .23 to .63 for seven qualities independently assessed by two raters.
- Edwards et al.3 report results from several meta-analyses showing inter-rater reliabilities ranging from .52 to .96, with a median of .83. Reliabilities for studies using structured interviews ranged between .82 and .84, and for those using unstructured interviews, reliabilities ranged from .61 to .75.
- Nowacek et al.4 reported inter-rater reliabilities for overall impressions of applicants that were .57 before the interview and .55 after the interview.
- Richards et al.13 reported an inter-rater reliability of .67 for panels of 13 interviewers.
- Van Susteren et al.14 reported an inter-rater kappa reliability of .79 for interviewers providing ratings scored within one point of each other on a five-point scale. Inter-rater reliabilities appear to be quite variable, but generally were higher (>.8) for structured interviews.
면접의 타당도 역시 그 결과가 모호하다.
The validity of interviews has also proven equivocal.
- Litton-Hawes et al.,15 analyzing 15 interviews using stimulated recall procedures from videotapes, found interviewers made inefficient use of time and focused on written materials to the detriment of exploring what they were intended to do. They advocated improved training of interviewers.
- Smith et al.16 compared first-year medical students' grades for two classes that had been interviewed with those of two classes that had not been interviewed (n = 44 and 79, respectively). Results showed no difference in grades.
- Perhaps the study producing the most compelling results in support of interview data for admissions comes from Powis et al.8 In a case–control study designed to retrospectively analyze differences between students who left medical school due to failure or withdrawal over a nine-year period and students who received honors, 56 paired cases (who left medical school) and controls (who completed medical school and who were matched according to gender, age, and entry cohort—all had excelled in their academic performances) were analyzed.
- Those who left had uniformly been rated more poorly in the interview, with effect sizes of..
- −4.17 for supportive and encouraging behavior,
- −3.46 for assessments of self-confidence and motivation to become a doctor,
- −3.11 for the overall rating,
- −2.76 for compatibility with study style of school,
- −1.98 for perseverance, and
- −.97 for tolerance of ambiguity.
- For differences between 58 pairs of students who graduated with honors and matched controls, honors graduates were rated more positively for...
- perseverance, ES = 2.98;
- self confidence, ES = 2.59;
- overall rating, ES = 2.17;
- tolerance of ambiguity, ES = 1.04; and
- supportive and encouraging behavior, ES = .86.
- For the remaining qualities, honors recipients received more positive evaluations (with ES < .40). The ES values in this study are meaningful and strongly suggest that interview ratings can discriminate between students who fail to complete medical school and those who complete medical school, as well as between those who graduate with honors and those who do not.
Meridith 등은 입학면접 도입을 강하게 지지하는 또 다른 근거를 제시한 바 있음. 여러 연구결과를 종합하면, 면접은 의학교육 중에서도 임상교육 단계에서의 학생 수행능력과 관련된 정보를 제공해준다.
- Meridith et al.6 also provided compelling evidence to support conducting admission interviews. They correlated data collected from the admission interview, as well as the MCAT score and undergraduate GPA, with National Board of Medical Examiners (NBME) Part II scores and subjective clinical assessments in pediatrics and internal medicine clerkships for third-year medical students. Admission interview data did not significantly correlate with NBME Part II scores, but did correlate with the subjective clinical assessments, accounting for over twice the variance as the next most potent predictor (interview = 10.4%, MCAT Science—Quantitative = 5.0%).
- Similar correlations of interview assessments with clinical assessments but not academic performances have been found in several studies involving non-medical health sciences programs.17–19
Thus, evidence exists that the interview provides information for admission related to students' performances in the clinical portions of medical education.
입학면접을 해야한다고 주장하는 여러 근거 중에서 의과대학이 사람간의 상호작용을 중요시하기 때문에 입학절차는 단순히 지원자의 서류와 성취도에 점수를 매기는 과정이 아니라 한 인간으로서, 미래의 동료로서 지원자의 특성을 평가하는 과정이며, 이는 특히 환자-의사 관계가 대단히 개인적인 것이기 때문이라는 언급을 찾을 수 없었다는 것은 특이하다. 면접은 비록 한시적으로 성적이나 수행능력이 저하되었더라도 그것이 가족의 죽음이나, 개인적 질병 등 다른 이유에 기인한 경우 이에 대한 공감을 표현할 수 있는 수단이기도 하다. 면접은 면접기관이 고도의 스트레스 상황이면서 고부담 결정과정에 있는 모든 사람에게 인간적인 손길을 내밀 수 있는 기회이기도 하다.
It is extraordinary that, among all the reasons given for conducting an admission interview, we found no mention of its use to demonstrate that a school values the personal interaction between human beings, that the admission process is not just a mechanical analysis of paper credentials and accomplishments but a judgment of one's qualities as a human being and a future colleague, particularly because the physician–patient relationship can be so intensely personal. The interview can also be a means of demonstrating compassion for applicants whose records may have temporary performance deficits that may be related to deaths in the family, illness, or other problems. The interview is a chance for an institution to place a human touch on what is a highly stressful, high-stakes decision process for all involved.
요약하자면, AAMC는 의과대학생 선발에서 인적특성을 보다 강조한다. 이러한 인적특성에 대한 강조는 미국의 소수 의과대학을 제외한 모든 의과대학에서 초점을 두고 있는 부분이며, 입학위원회는 학생선발시 면접에 큰 비중을 두나 어떤 것을 핵심 인적특성으로 할 것인가는 학교마다 다르다. 면접의 타당도에 대한 근거는 모호하지만, 면접평가점수가 임상평가를 예측할 수 있고, 면접점수가 낮을수록 유급이나 자퇴의 가능성이 높다는 근거도 있다. 면접의 신뢰도는 구조화를 통해 확보할 수 있다. 일부 연구자들은 인적특성 파악을 위해서는 면접보다 비용-효과적인 방법이 있을 것이라 주장한다.
To summarize, the AAMC has called for a greater emphasis on compelling personal characteristics in the selection of medical students. These compelling personal characteristics are the focal point of admission interviews that are conducted by all but a few medical schools in the United States. Evidence exists that admission committees give substantial weight to interview data in the selection of applicants, but what constitutes a compelling personal characteristic varies among institutions, with as many as 87 different qualities being considered for assessment. Even though the evidence for the validity of the interview has been equivocal, there is evidence that interview ratings are predictive of subjective clinical assessments, and low interview assessments are predictive of failure or withdrawal from medical school. The reliability of the interview can be improved using structured approaches.3 Some have argued there are more cost-effective methods than the interview for assessing compelling personal characteristics.
CONSIDERING ALTERNATIVES
면접은 지원자와 기관 모두에게 비용이 많이 드는 일이기 때문에, 그 대안에 대해서 이야기해보고자 한다. 여기서의 목표는 주요 인적특성을 측정하는 어려움을 분석하고 대안을 제시하는 것이다.
Because interviews are expensive for both the applicant and the institution, it is this issue we address in the remainder of this article. Our goal is to analyze the challenges in measuring compelling personal characteristics and then offer some practical and some perhaps less practical alternatives.
MCAT과 GPA의 의존도를 줄이는 것의 어려움
Challenges to Reducing Reliance on MCATs and Undergraduate GPAs
Cohen의 조언을 받아들이고자 할 때 직면하게 되는 문제점은 이기주의, 관성, 철학적/역사적 요인 등이 있다.
The major challenges facing any school adopting Dr. Cohen's recommendation to use a minimum GPA and MCAT score as a threshold and measures of compelling personal characteristics for admission are self-interest, inertia, and philosophical and historical factors.
> 의과대학의 이익 Self interest
입학에서 학업능력의 중요도를 줄이려는 시도시 의과대학이 마주하게 되는 가장 큰 장벽은 다른 사람들의 인식에 미치는 영향에 관한 것이다. 평균 MCAT점수와 GPA는 '최고의' 의과대학 순위 산정에 주요하게 쓰인다. 문제를 더 복잡하게 하는 것은 MCAT점수가 USMLE Step 1 점수와 상당한 상관관계를 가진다는 몇몇 연구의 결과이다. 상관관계가 인과관계를 의미하는 것은 아니며 제 3의 요인에 의한 것일 수 있지만 MCAT점수 비중을 줄이는 것은 USMLE Step 1 점수의 하락으로 이어질 수 있다. 탈락률이 높아지지는 않더라도 Step 1점수가 낮아지는 것 자체가 의과대학에 상당한 타격이다. 우리의 경험에 따르면 상위권 지원자는 USMLE Step 1 점수의 평균이 어떻게 되는지 문의해오는 경우가 많아, 적어도 표면적으로는 어떤 의과대학을 지원할지 결정할 때 이것을 고려하는 것으로 보인다. 따라서 Step 1 성적이 하락하는 것은 학생 모집에 악영향을 미칠 수 있다. 따라서 의과대학의 질을 평가하는 다양한 성과의 차원에서 이전 학업성적의 커트라인을 낮추는 것은 의과대학의 이익에 반하는 것이 될 수 있다. 반대로 높은 커트라인을 가져갈 경우 인적특성이 우수한 학생 찾아내려는 노력을 수포로 돌아가게 할 수도 있다.
Perhaps the biggest challenge a medical school faces in reducing reliance on academic credentials in admissions is the impact such a reduction may have on the perceptions of others. Mean MCAT scores and undergraduate GPAs are used as part of the formula in determining the “best” medical schools by U.S. News and World Report. Further complicating the situation, several studies20,21 have reported that MCAT scores correlate fairly strongly with United States Medical Licensure Examination (USMLE) Step 1 scores (multiple correlations of MCAT scores with NBME Part I, predecessor of Step 1, between .39 and .63, median = .5820; multiple correlation = .5921). Since a correlation can reflect either a cause-and-effect relationship (unlikely in this case) or the influence of a third variable (say academic or test-taking aptitude), a reduction in MCAT scores may put USMLE Step 1 scores at risk. Even if failure rates do not rise, lowered mean USMLE Step 1 scores can have substantial damaging effects on an institution. In our experience, top applicants commonly ask us for our mean USMLE Step 1 score, ostensibly a factor they are considering in making their medical school selections. Thus, a lower USMLE Step 1 mean score has the potential to damage recruitment efforts. Compounding the problem, some of the most competitive residency programs consider USMLE Step 1 scores in their decisions. Thus, from the standpoint of various outcomes used to assess the quality of medical schools, ignoring academic credentials beyond a low threshold will bump up against self-interest. On the other hand, setting a high threshold may cripple efforts to identify students with the compelling personal characteristics that may be most prized.
> 관성 Inertia
입학과정은 고부담의 과정이며, 엄청나게 큰 사업들이 다방면에 걸쳐있다. 비록 변화라는게 쉽지 않은 것이지만, 이러한 상황에서는 더욱 어렵다. '합리적인 커트라인'이 어느정도인가에 대한 합의를 이루는 것만에도 상당한 논쟁이 필요할 것이다. 우리는 입학위원회가 받아들일 수 있는 커트라인에 도달하기까지 우리는 12년에 걸친 기간의 데이터를 가지고 다양한 커트라인을 시험해보았고, USMLE Step 1, Step 2, 의과대학 졸업에 미치는 영향을 검토해보았다. 커트라인은 GPA와 MCAT점수를 더 높이더라도 Step 1과 Step 2, 졸업에 영향을 주지 않는 선에서 결정되었다. 또 다른 관성은 입학절차에 대한 것인데, 입학절차의 관리요원은 그 직무가 만만친 않고 그들의 성취에 대한 자부심이 크다. 현재의 입학시스템에서 과도한 업무를 하고 있다고 느끼는 관리요원들에게 기존의 방식과 다르게 하라고 요구하는 것은 큰 부담이 될 수 있다. 따라서 교수들의 동의를 얻는 것 뿐만 아니라 관리요원의 지지를 이끌어내는 것 모두가 중요하다.
Admission is a high stakes, big business operation involving a large number of very eclectic individuals. Although change never comes easily, it is especially difficult under these circumstances. Coming to agreement about what constitutes a reasonable threshold will take a substantial and compelling argument. To arrive at threshold values that were acceptable to our admission committee, we analyzed performance data over a 12-year period in which we simulated various thresholds and the resulting impacts on the likelihood of first-time USMLE Step 1 and Step 2 passage and medical school graduation. Thresholds were adopted for which the likelihood of USMLE Step 1 and Step 2 passage and graduation did not improve with higher GPAs and MCAT scores.22 Other inertia lies with changing the process by which admission occurs. Admission staff have a tough job and take pride in their accomplishments. For staff feeling overworked under the current admission system, doing things differently could seem overwhelming. They may also feel personally threatened by change. Thus, it takes a concerted effort not only to get faculty buy-in but also to ensure that the administrative staff supports the changes.
> 철학적/역사적 요인 Philosophical and historical factors
마지막으로 철학적, 역사적 요인이 주요 장애물이 될 것이다. 일부 교수들은 학업적 측면에서 '최고로 우수한' 학생만 받아야 한다고 믿는다. 이런 상황에서 커트라인 아래에 있는 학생을 대상으로 비학업적 측정에 의지하는 것은 절대 못 할 일이다. 또한 학업능력이 떨어지는 몇몇의 학생의 역사는 변화의 노력에 부정적으로 작용한다. 한두명의 그러한 학생이 들어오는 것 만으로도 교수들에게는 추가적인 위험으로 간주되어 저항을 초래한다. 이러한 역사적, 철학적 차이를 극복하는 방법은 다양한 위험요인별로 학생의 수행능력에 대한 자료를 축적하는 것 뿐이다. 이를 통해서 한두명의 안 좋은 결과도 30~40명의 좋은 결과로 극복될 수 있다. 적절한 자료가 있지만 결과가 충분히 긍정적이지 않다면 위험과 이익을 비교할 준비를 해야 한다.
Finally, philosophical and historical factors are likely to be major obstacles. Some faculty believe that we should admit only the “best and the brightest” by academic measures. Reliance on non-academic measures beyond a low threshold would be an anathema from this perspective. The related issue is that a history of encountering problems with students who have low academic credentials can come back to haunt any effort to change. All it takes is one or two such students admitted under the new system encountering major academic problems for faculty to develop resistance to assuming additional risk. The only way to counter these historical and philosophical differences is to collect data on the performances of students in various risk categories (if there are such data available). In this way, the poor outcomes with one or two students can be put in perspective if there have been good outcomes with 30 or 40. If there are appropriate data and the outcomes have not been compellingly positive, one must be prepared to assess the risk and the benefits.
핵심 인적특성 평가의 과제
Challenges in Measuring Compelling Personal Characteristics
'커트라인 접근법'을 활용하여 인적특성을 기반으로 입학여부를 결정하기로 정해지면 그 다음 문제는 신뢰도와 타당도를 갖추어 이들 특성을 측정하는 것이다. 다음의 것들을 결정해야 한다.
If a faculty decides that it is willing to take the risk of using a threshold approach for screening applicants and then admitting them on the basis of compelling personal characteristics, it still faces the daunting task of reliably and validly measuring these qualities. Among the challenges are determining:
- What constitutes a compelling personal characteristic, and which is/are most compelling?
- What is/are best method(s) of measuring these qualities?
- To what extent are these qualities influenced by nature, nurture, or maturation?
- What are the costs of measuring these qualities?
- What are the ways of overcoming cunning adversaries?
핵심 인적특성에는 무엇이 있으며, 그 중 가장 중요한 것은 무엇인가?
What constitutes a compelling personal characteristic, and which is/are most compelling?
다른 것보다 더 중요한 핵심 인적특성은 무엇일까? 여러 문헌에서 면접 또는 다른 방법으로 평가해온 특성들을 다루고 있지만, 여전히 무엇이 가장 중요한가에 대해서 연구해볼 여지가 있다. Price 등이 주장한 성공한 의사의 87개의 특징은 좋은 시작점이 될 수 있으나, 그 숫자가 너무 많고 측정하기에 현실적이지 않다. AAMC에서 1970년대 초반에 활용한 방식이 도움이 될 수 있다. Price 등의 연구결과에 근거하여 NCWG는 Jack Collwell의 주도하에 7개의 인적특성에 대한 측정을 MCAT에 포함시켰다. (compassion, coping capabilities, decision making, interprofessional relations, realistic self-appraisal, sensitivity in interpersonal relations, and staying power—physical and motivational.) 이 작업팀의 권고가 실제로 활용된 적은 없지만, 이들이 밝힌 인적특성 집합은 의과대학 학생선발시 가장 중요한 인적특성이 무엇인가를 평가하는데 도움을 줄 수 있다. 이 권고안이 발표된 이후 25년간 많은 일이 있었다. 인적특성이 오늘날에도 동일한가를 단정지을수는 없다. 다양한 방법으로 이 권고안을 업데이트 할 수는 있겠지만, 여전히 지역 특성을 반영하기 위해서는 각 기관이 권고안을 자체적으로 다시 평가할 필요가 있다. 그럼에도 국가적으로 정의된 것이 있다면 각 기관이 개발하는 것 보다 좋을 것이다.
What are these compelling personal characteristics that might trump other indicators? Although the literature offers insights into some qualities that have been assessed by interview and other means, there clearly is room for research into what are the most salient qualities. The 87 positive qualities of successful physicians identified by Price et al.23 might be a good starting point, but that number of qualities makes it a daunting starting point and, most likely, impractical for measuring. An effort to improve assessment methods for prospective medical students by the AAMC in the early 1970s might be of some help in this regard. Based upon the work of Price et al.,23 the Non-Cognitive Working Group, under the leadership of Jack Collwell, proposed specific objective measures of seven personal qualities be incorporated into the MCAT: compassion, coping capabilities, decision making, interprofessional relations, realistic self-appraisal, sensitivity in interpersonal relations, and staying power—physical and motivational.24 Although the recommendations from this working group were never acted upon, the set of personal qualities they identified might contribute usefully to the dialogue about the most salient personal characteristics to assess during the medical school selection process. A lot has transpired in the quarter century since these recommendations. Whether the same personal qualities would be identified today cannot be determined. It might be worth convening a similar working group to update the work or, perhaps, use the nationally directed multi-institutional process employed for the Medical School Objectives Project (MSOP) to update the recommendations. Whatever the recommendations would be, they would still need to be assessed by each institution for local relevance. However, it would be of substantial help to have a nationally defined set as a starting point rather than having each institution develop its own.
이 인적특성을 측정하는 최선의 방법은 무엇인가?
What is/are the best method(s) of measuring these qualities?
여기에는 두 가지 어려움이 있다. 첫째로 인적특성을 측정하는 것은 이를 측정가능한 것으로 정의하는 단계가 필요하고, 이를 위해서는 인적특성을 어떤 '행동'으로 정의해야 할 뿐만 아니라 그 행동을 보았을 때 대부분의 사람들이 어떤 인적특성을 반영한 것인지에 인식할 수 있을 정도로 합당해야 한다. 이는 쉬운 일이 아닌데, 이타성을 예로 보자. MSOP는 이타성의 7개 특성을 규정하고 있는데, 그 중 하나는 "자신의 지식, 임상술기의 한계를 인지하고 지속적으로 그 능력을 개발하는 것"이다. 이는 보는 사람에 따라서 "이타성"의 범주에 들어가지 않을 수도 있다. 그리고 이것이 이타성으로 인정된다 하더라도, 대부분의 사람들이 이를 측정하기 위한 평가법을 개발해야 한다.
Measuring compelling personal characteristics is challenging for at least two reasons. First, measuring a personal quality requires the difficult step of defining the personal quality in measurable terms. This involves defining the personal characteristic not only in behavioral terms but also in behavioral terms that most reasonable people would recognize as reflecting the personal quality if they were to see it. This is not an easy thing to do. Take altruism as an example. The MSOP delineated seven qualities of altruism that medical students must demonstrate before graduation to the satisfaction of the faculty. The last of these qualities is “The capacity to recognize and accept limitations in one's knowledge and clinical skills, and a commitment to continuously improve one's knowledge and ability.”25 This is not one of those components of altruism that might be in everyone's definition of altruism. Even if it were a generally agreed upon element of altruism, developing an assessment method whereby most people would recognize it if they saw it offers multiple challenges.
두 번째는 거의 모든 무한히 다양한 상황에 일관되게 나타나는 인적특성을 찾는 것의 어려움인데, 다시 한번 이타성을 예로 들어보자. 대부분의 입학위원회는 고부담의 면접상황에서만 점수를 잘 받기 위하여 이타성을 보여주는 지원자를 원하지는 않을 것이다. 그러나 대부분의 입학위원회는 지원자의 경력을 보거나 면접에서의 행동을 보는 것 외에 더 이상 할 수 있는게 별로 없다. 추천서에서 약간의 정보를 더 얻을 수 있을지는 모르나, 반드시 그런 것은 아니다. 또한 추천서 작성자는 지원자에 의해서 결정되므로 추천서에 쓰여진 내용을 얼마나 신뢰해야 하는가에 대한 문제가 있다.
The second reason is that one is most interested in stable qualities that have a high probability of occurrence in an almost infinite number of different situations. Take the example of altruism again. Most admission committees would not be particularly interested in an applicant who showed signs of altruism only in a high-stakes interview situation or in the period when the incentive to get into medical school provides the motivation for volunteering in various ways. However, most admission committees have little more to go on than an applicant's record of volunteering and his or her performance in an interview. Letters of recommendation may provide some additional information, but not necessarily. Further, because the letter writers are chosen by the applicant, it is difficult to know how much confidence to place in the information contained in the letters.
입학위원회는 제한된 정보 속에서도 지원자가 의사가 된 이후에도 환자와 지역사회를 우선시할 것인가에까지 예측하고자 노력한다. 의사에게 있어 이타성은 여러가지 다른 방식으로 나타나는데, 의뢰를 하는 동료의사와의 사업적 관계, 투자의 이해상충, 제약회사에 의한 인센티브, 봉사활동 등등 다양하다. 이 목록은 끝도 없고 이해상충의 가능성은 널려있다. 입학절차에서 관심을 두는 것은 하나의 구체적인 유혹뿐만 아니라, 대부분의 상황에서도 이타심을 유지할 것인가에 대한 것이다. 지원자가 평가자가 듣고 싶어하는 말을 하는 것, 누군가에게 조언을 받은 대로 대답하는 것, 진실로 이타적인 모습을 가진 것을 구분할 수 있을까?
The admission committee must work within the confines of the information it has, but it is trying to project whether the applicant will always and foremost put the patients' and community's needs above his or her own when he or she becomes a physician. This selflessness manifests itself in the practicing physician in many different ways, from business relationships with referral partners, conflicts of interest from investments, responses to incentives provided by drug and equipment manufacturers, willingness to volunteer for free medical clinics, etc. The list is endless and the potential for conflicts of interest is similarly pervasive. In the admission process, the interest is not just in whether the applicant will succumb to one specific temptation, but whether he or she will choose altruism as a guiding principle in most situations, if not all. During an interview, for example, how does one separate an honest portrayal of an applicant's response to a hypothetical situation from a carefully crafted response the applicant thinks the interviewer wants to hear, or perhaps that the applicant has been coached to provide by a preparation service?
이러한 인적특성은 천성, 양육, 성숙에 얼마나 영향을 받을까?
To what extent are these qualities influenced by nature, nurture, or maturation?
인적특성에 대한 또 다른 이슈는 그것이 과연 '안정적인' 것인가 하느냐는 점이다. 정직성과 같은 인적특성이 Bullimore가 말한 것과 같이 18세까지 완전히 결정되는 것인가? 정말로 '교육불가능한' 자질인가? 만약 교육불가능하다면, 그 가치를 중요시하는 집단 내에서 변화할 가능성은 없는가? 의과대학생이 의대를 다니는 동안 변화한다는 연구는 매우 많다. Rezler는 의과대학생에 대한 연구를 통해 의과대학환경이 학생들의 휴머니즘 수준 저하에 큰 영향을 미치며, 교수들의 태도/기술/헌신이 변하지 않는다면 교육과정변화로는 더 인간적인 의사를 만들 수 없다고 주장한다. Bland는 일차의료관련 전공 선택을 다룬 연구에서 Rezler의 관점을 다시 한번 강조하면서 의과대학기간 내에 휴머니즘 수준이 감소하는 것에 있어서 의학교육의 영향을 논하였다. 만약 의과대학이 이렇게 부정적인 영향을 준다면 의과대학문화를 변화시켜서 긍정적인 인적특성을 촉진할 수 있다는 주장이 가능하다. 이타성을 중시하고 이타성을 보여주는 교수들이 있다면 학생들도 이타적인 의사가 될 것이다. 또한 만약 사람들이 서로 다른 속도로 성숙한다면, 초기 단계에 있는 사람들은 이런 문화에 의해 더 많은 영향을 받을 것이다.
Another issue pertains to whether personal qualities are truly stable. Do personality traits such as honesty truly solidify by age 18 as argued by Bullimore?12 Are these truly “non-teachable” traits? If they are not teachable, are they malleable through immersion in a culture that values these traits? There seems to be considerable evidence that medical students change during medical school. Rezler,26 in a literature review on medical students' attitude changes during medical school, stated that the medical school environment was largely responsible for the decreasing humanism in medical students and that curricular innovations are unlikely to result in more caring doctors until a majority of medical teachers model these necessary attitudes, skills, and dedication. Bland et al.,27 in a comprehensive review of the literature on the determinants of primary care specialty choice, echoed Rezler's views on the decline in humanism during medical school and the negative influence of medical education. If medical school can have such a profoundly negative effect on students' humanism, it does not seem too far-fetched to suggest that a properly focused medical school culture could promote positive personality characteristics. With a culture that values altruism and faculty who demonstrate altruism always and foremost, it is conceivable that students might be nurtured into becoming altruistic physicians. Further, if people mature at different rates, those at earlier stages of development may be even more likely to be affected by an altruistic culture.
이 이슈는 제법 복잡하다. 만약 개개인의 인적특성이 의과대학 기간 내에 성숙한다면, 입학절차에서 해결할 문제는 '바람직한 방향'으로 성숙할 가능성이 있는 학생을 뽑는 것이다. 만약 인적특성이 변하지 않는다면, 입학절차에서 해결할 것은 신뢰도와 타당도를 높여서 이런 인적특성을 측정하고 이를 충분히 고려하는 것이다. 문제는 둘 중 하나가 절대적으로 옳지 않다는 점이다. 어떤 인적특성은 말랑말랑한 반면 다른 인적특성은 거의 변하지 않는다. 문제를 더 복잡하게 만드는 것은 말랑말랑한 특성과 변하지 않는 특성이 개개인마다 다르다는 점이다. 이 이슈를 논하는 것은 아마도 가장 어려운 문제 중 하나일 것이다.
The implications of this issue are particularly complex. If individuals do mature in their personal qualities such as altruism as they progress through medical school, the challenge for the admission process is to identify those who are most likely to mature in desirable ways. If personal qualities are stable, then the challenge for the admission process is to develop reliable and valid measures of these qualities and then to give them appropriate consideration. The problem may be that it is not an either/or proposition. Some personal qualities may be relatively malleable while others may be relatively stable by the time students enter medical school. To make matters even more complex, qualities that are malleable and those that are stable may vary among individuals. Sorting out this issue may be one of the greatest challenges to developing effective measures of personal qualities for medical school admissions.
이들 특성을 평가하는데 드는 비용은?
What are the costs of measuring these qualities?
입학절차는 지원자와 기관 모두에 큰 비용이 드는 과정이며, 추가적인 측정은 더 많은 비용을 수반한다. 기존에 수집된 정보로부터 새로운 수치를 이끌어내든, 더 좋은 방법으로 측정하든 문제는 비용이다. 측정하려는 측성이 많을수록 비용도 커지고, 새로운 정보는 새로운 측정법을 사용하기 전에 비해서 더 많은 가치를 제공할 수 있어야 한다.
The admission process is expensive for both the applicant and the institution, and adding measurements of new qualities in a rigorous manner will add more costs. Even if one uses information that is currently collected and derives new measures from it, or measures the qualities in a more rigorous manner, the change will add costs. The larger the number of personal qualities measured, the greater the costs incurred in their measurement. The new information must provide something of value beyond what was available before the new measures were added.
교묘한 지원자를 가려내는 방법?
What are the ways of overcoming cunning adversaries?
가장 큰 문제 중 하나는 지원자와 지원준비 서비스의 눈속임 방법이다. 의과대학의 일부 지원자는 카멜레온과 같아서 순간적으로 테레사수녀가 되기도 한다. 또한 일부 전문적 서비스에서는 그 하루를 위해서 좋은 특성을 보여주는 방법을 조언해준다. 잘 만들어진 소설 속에서 진실을 찾아내는 것이 중요하다.
One of the greatest challenges facing any effort to systematically measure personal qualities will be the cunning ability of applicants and preparation services. Some applicants to medical school seem to have a chameleon-like ability to adopt the short-term personality of “Mother Theresa” and the career interest du jour. Further, the survival of some professional services depends on their ability to help applicants with repelling personal characteristics display compelling ones for a day. Sorting out fact from carefully crafted fiction will make developing a standardized measure of compelling personal characteristics a difficult challenge.
가능한 도구들
Some Possible Measurement Approaches
만약 각 의과대학이 가장 비용이 덜 드는 방법을 원한다면 기존 정보의 새로운 활용이 좋을 것이다. AMCAS는 지원시 자기소개서와 에세이를 작성하도록 하고 있고, 이로부터 인적특성을 평가할 수 있다. 이미 대부분의 의과대학이 면접 시행하고 있어서 면접도 활용가능한 방식이다. 마지막으로 성적증명서, 부모교육이력, 재정정보 등도 인적특성에 대한 정보를 줄 수 있다. 예컨대 한 가족에서 처음으로 대학에 가는 지원자는 많은 의과대학에서 긍정적인 평가를 받는다.
If schools attempt to take a least-cost approach to measuring compelling personal characteristics, making new uses of old information, (i.e. information already available) will be helpful. The American Medical College Application Service (AMCAS) application's Personal Statement and Essay could reflect compelling personal characteristics. Because all but a few medical schools already interview applicants, the interview itself might be a readily available source of information regarding compelling personal characteristics. Letters of recommendation are also commonly required to supplement the AMCAS application. Finally, there are elements of the transcript, parent(s)' education, and financial data that might give insights into personal qualities. For example, an applicant who is the first in his or her family to go to college, let alone go to medical school, is often given positive consideration in many medical schools. Other such insights might be gained from a careful and thoughtful consideration of such information.
자기소개서와 에세이
Personal statement and essay
자기소개서는 대체로 형식이 없으며, AMCAS지원 중 작성하게 되는 것이다. 자기소개서를 살펴보면 매우 다양함을 알 수 있는데, 이를 통해 인적특성을 평가할 수 있다. 자기소개서는 손대지 않은 자원과 같다. 자기소개서 해석을 잘 하기 위해서는 지원자들이 자기소개서를 통해서 의료전문직에게 필요한 인적특성을 잘 보여줄 수 있다고 믿는 지, 그리고 자기소개서가 다른 사람을 포함한 'group project'인지를 아는 것이 중요하다. 이 문제를 평가하기 위해서 우리는 의과대학 1학년생을 대상으로 설문조사를 했다. 3년간의 조사 결과를 보면 53~84%의 응답자가 자기소개서가 자신의 인적특성을 잘 보여준다고 생각하는 것으로 나타났고, 41~44%의 응담자는 자기소개서에 다른 사람의 도움을 받았다고 하였으며, 그 중 15~51%의 응답자는 내용 개발에, 2~6%는 전문적 서비스의 도움을 받았다고 했다. 비록 자기소개서가 많은 의과대학생으로부터 인적특성을 잘 드러내는 것으로 인식된다고 하더라도, 준비과정에서 다른 사람의 도움을 받는 것은 모든 지원자에 대해서 그 정확성을 담보하는데 제한을 만든다. 더 나아가 형식이 없기 때문에 한 자기소개서에서 강조하는 인적특성은 다른 자기소개서에서 강조하는 것과 다를 수 있다. 비표준화된 정보를 가지고 타당성있는 비교를 하는 것은 어려운 문제이다.
The personal statement is a relatively free-form essay that the applicant produces as part of the AMCAS application. A review of a sampling of such statements leads us to say that these statements are extremely variable. A literature search yielded no citation of its being used or evaluated to assess personal characteristics of the applicant. This would seem to be an untapped resource. To properly interpret the personal statement, it is important to know whether applicants believe the personal statement allows them to accurately represent their personal characteristics that qualify them for the profession of medicine, and whether the personal statement is a “group project” involving input from various others. To assess these issues, we surveyed matriculating first-year medical students for three years.28 Across the three years, 53–84% of the respondents indicated that the personal statement adequately represented some element of their personal characteristics; 41–44% reported the personal statement involved input from others, with 15–51% reporting input in content development and 2–6% receiving input from professional services. Although the personal statement was considered by the large majority of matriculating medical students to adequately represent their personal characteristics, questions about help received in its preparation limit the confidence that admission committees can place in its accuracy for all applicants. Further, because of its free-form nature, any given personal statement will highlight a set of personal characteristics potentially different from the set highlighted in another applicant's personal statement. Making valid comparisons of applicants' personal characteristics from such non-standardized information offers significant challenges.
면접 Interview
면접은 지원자가 직접 캠퍼스에 모습을 드러내는 몇 안되는 순간이다. 이 중요한 순간을 어떻게 활용하고 어떤 목적으로 사용하는가가 중요하다. 인적특성 평가를 위한 면접의 가능성은 두세시간의 면접 그 이상이다. 지원자의 총 방문시간은 그 몇배이며, 오리엔테이션/투어/재학생과의 상호작용/만찬 등등이 모두 들어간다. 이 모든 시간동안의 관찰이 유용한 정보가 된다.
The interview is one of the few times, if any, prior to the admission decision when the applicant is physically present on the campus. Deciding how to use that precious time and to what purpose is a critical decision. The potential of the interview for assessing personal qualities extends beyond the two- or three-hour interview; the visit usually lasts at least twice that long and includes orientation activities, tours, interactions with current students, luncheons, etc. Observations of applicants during these other times might also provide useful information.
연구 결과를 살펴보면 구조화된 면접이 인적특성을 가장 잘 보여주는 것으로 나타난다. 구조화된 면접의 특징은 다음과 같다.
Research suggests that structured interviews yield the most reliable estimates of personal qualities. A structured interview's features can include
- selecting interview content based upon a job analysis (e.g., the critical-incidents technique),
- standardizing the questions asked of all applicants,
- providing interviewers with sample answers to questions to help them give consistent ratings, and
- having the interview conducted by a board or panel of interviewers.
- One critical additional feature is providing training for the interviewers. Structured interviews require at least a minimal introduction to interview protocol and a rating system.
또 다른 중요한 이슈는 면접관의 자격에 대한 것이다. Patrick은 비의학전공자들을 면접관으로 고용하여 훈련시켰다. Collins 등은 의대교수와 함께 소비자그룹의 대표와 교육분야 전문가를 면접 패널로 포함시켰다. 비-의대교수가 활용된 바 있지만 이들이 다른 평가경향을 가지는지에 대한 연구는 부족하다. 이러한 연구가 향후 필요할 것이며, 평가자의 특징과 지원자의 특징과의 상호작용에 대한 연구도 필요하다.
Collins et al.10 provided a half day of training for interviewers who were to be part of a 20-minute panel interview and then observed applicants interact in a group problem-solving session. Another potentially important issue concerns interviewers' qualifications. Patrick et al.5 hired non-medical people to serve as interviewers and trained them in the protocol. Collins et al.10 included representatives of consumer groups and experts in education along with medical school faculty on interview panels. Although non-faculty were used in these studies as interviewers, there was no effort to determine whether interviewers with different characteristics (consumer group representatives, education experts, medical school faculty) had distinguishable rating tendencies. Future research will need to determine to what degree interviewer characteristics produce detectable differences in interview results. There may also be an interaction between interviewer characteristics and applicant characteristics at play in the results.
Edwards 등은 구조화 면접의 특징을 정의하는 과정에서 면접관을 참여하게 했다. 한 명의 면접관을 참여시키는 것만으로도 평가결과의 신뢰도를 향상시켰으나, 다수의 면접관은 면접과정의 역학을 예측불가능하게 만들 가능성도 있다. 다수의 교수들이 한 명의 지원자를 평가하는 것은 위협적인데, 이는 비록 면접이 구조화되어있고 부드러운 분위기에서 진행되더라도 숫자의 불균형이 위협적으로 느껴질 수 있기 때문이다. 우리의 학생들을 대상으로 조사해본 결과, 패널식 면접을 하는 학교에 대해서 매우 비판적이었다. 여학생들이나 유색인종이 특히 비판적이었다. 면접 형식, 면접관의 특성, 지원자의 특성이 복합적으로 섞여서 상당한 영향을 줄 수도 있다.
Edwards et al.3 considered a panel of interviewers to be part of the defining characteristics of a structured interview process. Having more than one interviewer enhances the reliability of the resulting ratings, but multiple interviewers can impact the dynamic of the interview in potentially unpredictable ways. Having multiple faculty interrogate an applicant can seem threatening because, even if the interview is structured to be collegial, the imbalance in numbers can be intimidating. In interviews conducted at our institution about our students' experiences in the admission interviews they had experienced at various schools, the students have been very critical of schools using a panel approach. Women students and students of color have been especially critical of schools using panel interviews. The interview format, interviewer characteristics, and applicant characteristics may represent a complex mix of factors that could have a major impact on the admission interview process.
마지막은 면접의 특성과 관련된 것이다. Collins 등은 두 가지 종류의 면접을 보고한 바 있다. 하나는 20분짜리 구조화면접으로 1명의 지원자를 2명의 패널리스트가 면접하는 것이고, 다른 하나는 2명의 패널리스트가 50분의 집단토론에서 6명의 지원자를 보게 된다. 141명의 지원자 중에서 두 면접 종류의 일치도는 0.62였다. 비록 상당한 상관관계이고 통계적으로도 유의지하나 rating variance의 40% 이하만을 설명할 뿐이다. 이는 여전히 50%이상의 variance를 설명불가능한 영역으로 남겨두는 것이며, 면접의 특성과 어떻게 구조화되느냐가 면접 결과에 상당한 영향을 준다는 의미가 된다. 면접을 구조화하는 데 있어서 핵심은 평가하고자 하는 인적특성이 의미있는 방식으로 평가되느냐 하는 것이다. 어떤 인적특성은 평가하기에 용이하나 어떤 것은 그렇지 않다. 면접상황에서 다양한 인적특성을 어떠게 측정할 것인가에 대한 연구가 필요하다.
A final concern relates to the nature of the interview. Collins et al.10 reported two types of interviews being conducted: a 20-minute structured interview of one applicant by two panelists and two panelists observing six applicants as they participated in a 50-minute group exercise designed to stimulate debate. Separate panelists were used for the two types of interviews. Over 141 applicants, the correlation between ratings of the two interview types was .62. Although this is a relatively large correlation and was statistically significant beyond the .0001 level, it accounts for less that 40% of the rating variance. Disattenuating the correlation for the less-than-perfect reliability of the panel ratings (.67) still left over 50% of the variance in the two ratings unexplained, which clearly indicates that the nature of the activities and how they are structured for applicants may have a substantial effect on the results of the interview. The key is to structure the interview such that the personal qualities of interest can be assessed in a meaningful manner. Even if some personal qualities such as resourcefulness may be amenable to assessment in this type of situation, others such as altruism are likely to suffer from the artificiality of the conditions. Research related to how to measure various personal characteristics in an interview situation is clearly needed.
캠퍼스방문 중 비면접부분에서도 많은 것을 얻을 수 있다. 360도 평가 활용.
Much may be gained from the non-interview portion of the campus visit. The time applicants spend interacting with each other, participating in the orientation activities, meals, tours, etc. potentially can offer much insight into applicants' personal characteristics. One approach might be to adopt an element of the 360-degree evaluation model being explored for resident and physician evaluation by the Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS). In this approach, almost everyone who comes into contact with the individual being rated provides a rating. In the admission interview case, one could do the same for the applicants during the non-structured interview time. Medical students, receptionists, food-service workers, tour guides, bus drivers, dean's staff, and others who interact with the applicants during the non-interview activities could be asked to rate the applicants. Because a relatively large number of applicants appear at the same time, such an evaluation would have to be picture-coded. It would probably be unreasonable for all of these different types of individuals to rate all applicants on all of the desired personal characteristics. Their contact would be so variable and transitory that it would probably be mostly wasted effort. If, however, these different individuals reported only memorable interactions of both positive and negative kinds, the strategy might provide useful information. At the very least, such an approach might be worth exploring.
다양한 인적특성에 대해서 방문의 구조화된 부분과 비구조화된 부분으로 나누어 검사해볼 수도 있다. 변하지 않는 인적특성을 신뢰도와 타당도를 갖추어 측정하기 위해서는 관심과 창의성이 필요하다. 비침습적 검사를 만드는 것도 고려해볼 수 있으나 인간적으로, 그리고 편집증적이지 않게 해야한다. 인적특성을 측정하기 위한 시도를 하는 과정에서 그 부작용에 대해서 항상 신경써야 한다.
One could also build tests of various personal qualities into the structured and unstructured portions of the visit. To reliably and validly measure personal qualities that are stable across time and situation will take care and creativity. The potential for building unobtrusive tests into the interview visit might be worth exploring, but care must be taken that it is done humanely and does not create a climate inducive of paranoia where the applicant feels under the microscope at every moment. As attempts are made to measure personal characteristics, one needs to be mindful of the potential side effects that these measurements might produce.
추천서
Letters of support
추천서는 AMCAS 지원과정에서 흔히 요구되는 것이다. 추천인을 어떻게 선정하는가에 대해서 의과대학마다 기준이 다르고, 무엇을 기술해야 하는가에 대해서도 각기 다르다. 종종 해석이 어려운 경우가 있는데, 이는 추천인을 지원자가 결정하기 때문이며, 그 형식이 자유롭기 때문이다. 어떻게 추천서를 바탕으로 지원자를 평가해야하는가가 아주 명확한 것은 아니며, 만약 지원자를 정확히 보여준다면 한 지원자의 추천서를 다른 지원자의 것과 비교해야 할 것인가를 고민해야 한다.
Letters of recommendation are commonly requested as supplements to AMCAS application materials. Medical schools vary in how they instruct applicants to select letter writers and the degrees of structure imposed on what is to be written. The consequence is that letters of support are often difficult to interpret. Because the writers are chosen by the applicants and the formats are often relatively free-form, it is never clear how representative a given letter is of the applicant, and, if it is an accurate portrayal, how to evaluate the quality of one applicant's letters of support against those of another applicant.
더 문제가 되는 것은 거짓 정보가 기술된 추천서이다. 이는 이론적으로만 가능한 것이 아니고 우리는 실제로 입학한 학생 중 추천서를 날조한 학생을 퇴학시킨 적이 있다. 더 나아가 입학절차에 대한포커스그룹 미팅에서 일부 학생들은 추천서 작성에 관여했음을 인정한 바 있다.
Even more problematic is the risk of fraudulent letters of support. This is not just a theoretical possibility. We had to dismiss a student we had admitted to our medical school when it came to light that the letters of support were self-fabricated. Further, in focus-group meetings about the admission process, some students have admitted to participating in preparing some of the “letters of support.”
또 다른 문제는 기관에 따라서 추천서의 평가가 매우 달라진다는 것이다. 무난한 추천서가 매우 좋게 평가받을 수도 있고, 과장된 추천서가 매우 부정적으로 평가받을 수도 있다. 추천서 해석의 총체적 접근법이 중요하다. 추천서에 무엇이 들어가야 하는가에 대한 국가적 합의가 있다면 평가가 용이할 것이다. 더 나아가 내용분석의 절차를 활용하는 방법도 있다. 거짓 추천서에 관해서는 표준화된 형식을 활용할 수 있고, NIH에서 현재 사용중인 전자추천서시스템을 활용할 수도 있다.
Another challenge is that interpreting letters of support varies and often depends upon previous institutional experience with individual letter writers. A relatively bland letter from one writer might be considered extremely positive, and a comparable letter from a more effusive letter writer could be considered extremely negative. A more systematic approach to interpreting letters of support would be helpful. If national standards were to be developed about what should be included in letters of support for medical school, at least the content of the letters might be easier to evaluate. Further, there may be ways of applying content analysis procedures to letters of support that could aid in their interpretation. To address the concern about fraudulent letters, if a standard format could be adopted, perhaps an electronic system like that currently required by the National Institutes of Health (which has built-in security factors) could be used. These issues deserve further consideration and research.
성적증명서, 부모의 교육수준, 재정자료
Elements of the transcript, parent(s)' education, and financial data
AMCAS파일, 지원서, 다양한 인구통계적 특성으로부터 핵심 인적특성을 뽑아내는 것은 복잡한 과정이다. 일부 주요 마커들을 유심히 볼 필요가 있는데, 인종/첫 대학진학자/농촌지역출신자/낮은 수입 등이다. 이를 봄으로써 취약계층 출신에게 가산점을 줄 수도 있다. 이러한 분석은 종종 정치적 이슈가 되기도 한다. 예컨대 의과대학에 가기 위해서 많은 역경을 겪은 지원자는 겨우 생계를 유지할 수 있는 정도의 삶을 살아왔을 수 있다. 의과대학이 요구하는 추가적인 것들은 그의 삶을 무척 힘들게 만들 수도 있다. 우리의 경험상 차이를 유발하는 삶의 역경을 이겨내는 것은 학업적 능력이 아닌 경우가 많다. 이러한 이슈가 종종 면접에서 논의되나 학업적, 인구통계학적 프로필을 통해서 고위험과 저위험을 구분하는 방법이 있을 것이다.
Deducing compelling personal characteristics from the AMCAS file, supplemental application form, and various demographics is a complex task. Certain key markers are sometimes used to consider applicants for scrutiny, such as ethnicity, first generation to attend college, rural/inner city residence, low income, etc. The general purpose of interpreting these markers is to give applicants from underrepresented and disadvantaged backgrounds consideration for admission in the context of their backgrounds. Such analyses are challenging and often become politically charged issues. For example, an applicant who has overcome substantial adversity to make it to medical school may be barely keeping his or her head above water under the weight of life's demands, obligations, or lingering effects of earlier obstacles (single parenthood, emotional trauma from loss of relatives/friends, surmounting a poor early educational system, etc.). The additional demands that medical school imposes can often push him or her under. Determining the difference between an applicant who can make it and one who cannot is difficult. In our experience, it often is not academic ability but the crush of life's obligations that makes the difference. This kind of issue is often discussed in the interview, but there may be ways to separate high risk from lesser risk through comparing the academic and demographic profiles of students who have made it through medical school with those who have not. More research in this area could yield valuable information.
A PROPOSAL FOR A UNIFIED SYSTEM OF ASSESSMENT
There are larger forces in the universe of medical education that might be usefully applied to assessment in the admission process. MSOP has identified four major objectives for medical education, each of which has six to 11 subobjectives. The ACGME and the ABMS have jointly identified six competencies for the practice of medicine. The MSOP competencies map fairly closely on these six competencies. Discussion of extending the ABMS/ACGME competencies into undergraduate medical education were held at the 2002 meeting of the Central Group on Educational Affairs, as was discussion of the possibility of integrating student-evaluation methods across undergraduate medical education (UME) and graduate medical education (GME).29
The potential benefits of integrating assessment methods across UME and GME would seem to be many. The ABMS and ACGME are beginning work on operationally defining how to measure the six competencies, beginning with communication skills. One of many tools that comprise the toolbox being developed for this purpose is the 360-degree evaluation. For a resident physician who is being evaluated, for example, one could have supervising physicians, nurses, patients, and administrative staff complete evaluations. Cost-effective methods of obtaining these evaluations are being developed.
Clearly, competencies appropriate for physician recertification would be more advanced than those for resident certification to practice, and, similarly, medical student competencies would be less advanced than those for residents. If we extend the concept of integration into the admission process, the competencies identified for applicants to medical school (pre-medical competencies) would be more rudimentary than those established for medical students. However, if one considers competencies to be a continuum from cradle to grave, the natural progression could serve as a means for assessing individuals at specific defining points. The evaluation methods used could build upon one another for continuity so that students feel a sense of progression and are better able to self-regulate their learning (monitor progress, identify learning needs, and adjust study accordingly). This Unified System of Assessment would enable all parties interested in measuring the competencies of physicians and physicians-in-training to pool their resources and adopt a developmental approach to the measurement process.
Even now, some of the methods being developed by the ACGME/ABMS collaboration might have potential application to the admission process. As it progresses, the work of these groups may help to narrow the field of personal qualities that are of the highest priority for assessment. For example, the ACGME/ABMS collaboration has adopted the American Board of Internal Medicine (ABIM) process for peer and patient assessment. The ABIM recertification process involves having diplomates arrange for ten professional colleagues and 25 patients to answer ten questions about their overall medical care and communication skills.30 They use a computer-administered telephone survey to collect the information. Using a similar method, it might be that a reasonably small number of applicant-nominated individuals completing a telephone survey can provide reliable and valid assessments of an applicant's personal qualities. If it is found that other personal characteristics can be better assessed via interview, the interview could be better focused to provide more reliable and valid measures of these other characteristics. It might even be possible to add a SAMS-type interview to the MCAT administration. This would potentially make assessing personal qualities during the campus visit optional or it could emphasize the elements unique to the particular school.
The segregation of UME, GME, CME and recertification has gone on for far too long. We need to consider the entire process as a continuum that includes even the selection of students for medical school and the pre-medical requirements. Pooling the resources of the entire system devoted to the education, testing, certification, and recertification of physicians would contribute to making much more headway than can be done with the current fragmented and separate small-scale efforts. Developing a continuum of competencies is a first step, developing a unified system for assessment would be the next.
Assessing personal qualities in medical school admissions.
Abstract
'Articles (Medical Education) > 입학, 선발(Admission and Selection)' 카테고리의 다른 글
| 다면인적성면접: 같은 개념, 다른 접근법(Medical Education, 2014) (0) | 2015.02.16 |
|---|---|
| 의과대학에서의 성공과 관련된 요인들: 체계적 문헌고찰(BMJ, 2002) (0) | 2015.02.06 |
| 의과대학에서의 입학면접 (KMER, 2010) (0) | 2014.12.18 |
| 의과대학 선발에서의 질적 요소 (Acad Med, 1990) (0) | 2014.10.30 |
| 의과대학 선발에 대한 여러 관점 (Acad Med, 1990) (0) | 2014.10.30 |