의과대학 학생 선발을 더 개선할 수는 없을까?

Can we improve on how we select medical students?

Patricia Hughes, MSc FRCPsych

Admissions Office, Hunter Wing, St George's Hospital Medical School, London SW17 0RE, UK

E-mail: p.hughes@shgms.ac.uk

 

합당한 의과대학 입학정책을 운영하는 것은 좋은 의사가 될 잠재력을 가진 사람을 선발함으로서 사회에 대한 타당성(fair)을 갖춰야 하며, 또한 지원자에 대한 공정함(fair)을 갖춰야 한다. 선발은 엄밀한 과학은 아니지만, 모든 측면에서 최선을 다하기 위해 활용가능한 근거를 최대한 활용해야 한다. 단순한 학업적 성취 외에도 더 넓은 범위의 준거를 활용해야 한다는 것에 대한 폭넓은 동의가 있으나, 현실적으로 많은 의과대학이 다른 고려사항보다도 입학 전 학업성적을 가장 중요한 준거로 삼는다. 

그러나 학교 성적을 지적 역량의 척도로 활용하는 것에 대한 단점이 있어서, 대표적으로 A학점을 받는 것에 가장 중요하게 작용하는 요인이 사회적 계층이며, 개인의 능력과는 무관하다는 연구가 있다. 또한 의과대학생이 되려고 하는 학생들이 과학 과목에 집중하는 이유가 physical sciences 부분에서 인문 과목보다 더 좋은 점수를 받기 쉽기 때문이다. 시험 결과만이 타당하고 신되도 있는 자료라는 것은 '매력적이지만 오류가 많은' 신념이다. 우리는 모든 선발 도구들이 주관적 판단에 의존하고 있다는 것을 명심해야 하며, 각각의 도구들은 논리, 공정함, 공공의 검토(reason, fairness and public scrutiny)라는 규칙을 따라야 한다. 그러나 우리가 비인지적 준거를 고려하자고 하는 순간 의학의 여러 전문과목들은 다양한 skill을 필요로 하며, 따라서 그 준거들이 너무 협소해서는 안된다는 타당한 우려를 갖게 된다. 또한 우리가 비인지적 특징을 평가에 포함시키고자 한다면, 그렇게 이루어진 평가가 몇 년이 지난 후에도 개인의 특성을 잘 예측할 것이라는 확신을 갖고 싶어한다.

Getting the right policy for admission to medical school is a balancing act: be fair to society by choosing people with the potential to be good doctors; and be fair to the applicants—that diverse group of people who for many reasons want to set out on the long road to a medical career. Selection is not an exact science but we must use what evidence we have to ensure that we do our best by all concerned. There is widespread agreement that we should select future doctors on wider criteria than scores of academic success1, 2, though in practice many medical schools have valued pre-admission academic scores at the expense of other considerations3. There are recognized drawbacks to the use of school exam performance even as a measure of intellectual competence. One study has shown that a major causal determinant of A level results is social class, independent of ability4, and some would-be medical students elect to focus on sciences for their school leaving exams because very high marks are more easily achieved in the physical sciences than in the humanities5. The conviction that only exam results give valid and reliable data has been trenchantly dismissed as a ‘seductive but fallacious’ belief in the precision of quantitative tests6. We are reminded that all selective instruments depend on subjective judgments and each must be accountable to the rules of reason, fairness and public scrutiny7. However, if we decide to consider non-cognitive criteria, a legitimate concern is that the many specialties of medicine need diverse skills and they must not be too narrow. We also want to be reassured, if we include noncognitive characteristics, that we can assess them reliably and that such evaluation can predict personal character over years of practice.

의사에게 필요한 skill과 인성의 범위는 넓지만, 여전히 어떤 의사에게나 요구되는 특징이 있다. 충분한 지적 능력 외에도 정직성, 진실성, 양심 등이 좋은 진료의 중심에 있다. 도움을 주고자 하는 마음과 협력하려는 자세(Helpfulness and willingness to cooperate) 역시 중요하며, 환자들은 대인관계기술이나 공감능력이 뛰어난 의사를 좋아한다. 전문직으로서 개개인의 안녕(welfare)를 잘 유지하는 것 역시 중요하다. 의사들은 다른 직종보다 알콜중독, 약물남용, 자살 등에 취약하다. 탈진(burnout)역시 흔히 일어나는 일이며 이는 개인 뿐만 아니라 동료, 환자가 받게 되는 서비스의 질에도 큰 비용을 수반하는 것이다. 정신적으로 취약한 의사를 잘 지원해 주는 것이 하나의 답이 될 수 있을 것이지만, 더 중요한 것은 스트레스에 잘 대처할 수 있는 능력을 가진 사람을 애초에 뽑는 것일 것이다.

While we need to maintain diversity of skills and personality, there are some characteristics which we demand in any doctor. Enough intellectual ability to do the job, plus honesty, integrity and conscientiousness, must be at the heart of good practice8. Helpfulness and willingness to cooperate come close behind8, while patients give high priority to interpersonal skills and empathy2. The personal welfare of the profession is another consideration9. Doctors are more vulnerable than comparable professional groups to alcoholism, drug abuse and suicide10, 11. Burnout is well recognized, and has a high cost for the individual, for colleagues and for the quality of service that patients get12. One answer may be better support for psychologically vulnerable doctors12, 13 (together with improved working conditions for all doctors), but perhaps we should try to evaluate ability to deal with stress right from the start.

인성은 성년 이후에도 안정적으로 유지되는 특성인가?

ARE PERSONALITY CHARACTERISTICS STABLE OVER ADULT LIFE?

만약 우리가 의과대학생에게 원하는 인성을 찾고자 한다면, 이것이 과연 미래의 인성에 대해서 확실히 말해줄 수 있다는 자신감을 가질 수 있을까? 의과대학생과 이들을 15~30년간 추적한 연구에 따르면 middle age에 정신적으로 건강했던 의사들은 학생때에도 높은 자존감을 유지하고 있었고, 삶에 대해 열린/유연한 자세를 가지고 있었으며, 부모와 따뜻한 관계를 영유하고 있었고, 불안, 우울이 적었고 스트레스 상황에서 받는 화(anger)도 낮았다. 

If we seek to identify the personal characteristics we want in a medical student, can we have any confidence that they tell us anything about future personality or adjustment? Studies that assessed medical undergraduates and followed them up for between 15-30 years12, 14, 15 indicate that doctors who are psychologically well in middle age had good self esteem as students, had an open, flexible approach to life, enjoyed a warm relationship with their parents, and had little anxiety and depression and low anger under stress. 

반면, 후에 중년에 약물 오용, 자살, 탈진 등에 취약했던 의사들은 학생 때 역시 유의미하게 정신건강이 좋지 않았으며, 장기 연구에서 6년~45년의 간격을 두고 재평가(retest)를 했을 때 높은 test-retest correlation을 나타냈다. 이러한 열과는 인성의 연속적 특성이 행복하든 불행하든, 부유하든 가난하든 성년이 되어서도 유지되는 안정성(stable tendency)을 보임을 알려준다.

In contrast, doctors vulnerable to later substance abuse, to suicide and to burnout in middle age had significantly poorer measures of psychological health as undergraduates. Other long-term studies of stability of personality characteristics have shown that personality traits exhibit high test-retest correlations over intervals of 6 to 45 years16, 17, 18, 19. These findings signify a substantial continuity of personality disposition in adulthood, suggesting a stable tendency to be either happy or unhappy, well or poorly adjusted.

미래에 직무역량을 예측해주는 요인은 무엇인가?

WHAT FACTORS GENERALLY PREDICT FUTURE JOB PERFORMANCE?

의학과 의학 외 분야에서 모두 이런 것과 관련된 연구가 있다. 이러한 연구에 많은 돈을 투자한 산업계에서 유용한 정보들을 확인할 수 있다. 평균적으로 가장 생산성이 옾은 사람은 평균보다 40%쯤 더 잘벌고, 가장 나쁜 사람은 40%쯤 덜 번다는 결과를 보여주고 있다. 그런데 이것이 의학 분야와 많이 다를까? 주위를 둘러보면 최신의 지식을 갈고닦는 의사가 있고, 이런 것은 거의 하지 않는 의사가 있다. 이것이 유일한 criteria는 될 수 없지만 중요한 것임엔 틀림없다. 

There is relevant research both within medicine and outside it. Useful information comes from industry, where serious money has gone into finding out what makes a good professional20. They measure outcome in hard cash and find that the most productive people are about 40% better than average, while the least are 40% worse than average21. Is this too different from medicine to be relevant? Look around: we all know who gets the work done and keeps up to date, and who slips through life doing the minimum. These are not the only criteria for a decent doctor but they matter. 

복잡한 직무에 있어서 가장 훌륭한 예측인자는 mental ability에 대한 몇 가지 척도와 IQ이며, 더 높은 자리로 올라갈수록 IQ가 더 중요하다는 것이 지속적으로 나타나는 근거이다. 가장 높은 관리자 수준에서 전체 수행편차 중 70%가 이 것(IQ)때문이다. 따라서 높은 IQ가 중요하다는 것을 봤을 때 이것을 근거로 선발하는 것은 옳다고 할 수 있다. 예측력은 다른 몇 가지 요인을 추가하면 더욱 향상된다. 

There is consistent evidence that, for work involving complex tasks, the best predictor of effectiveness is some measure of mental ability or IQ, and the higher you go up the professional scale the more IQ matters. At the highest managerial level it accounts for almost 70% of performance variability22. So in demanding evidence of high IQ (even in the form of exam results) we have got something right. Predictability can be improved by including some measure of other factors. 

더 추가해야 할, 지속적으로 확인되는 요인은 '진실성(integrity)'와 성실성(conscientiousness)이다. 이것은 IQ와의 상관관계가 없다. 예측력을 높여주는 인자는 이것이 전부이다. 교육기간이 약간의 예측타당도를 높여줄 뿐이고, 얼마나 많은 과목을 들었는가는 아무런 관련이 없다. 이전 직장에서의 직무수행능력은 이미 직무를 수행하는 단계에 있는 사람에게는 관련이 있을지 몰라도, 시작하는 사람과는 관계가 없다. 이런 결과 중 몇 가지는 직관에 반하는 것이다. 왜냐하면 IQ가 다른 요인들과 중복되기 때문으로, 습득력이 빠른 사람은 이전 직장에서 좋은 수행능력을 보여줬을 것이지만, 이 자체가 이미 IQ와 높은 상관관계가 있기 때문에 예측타당도에는 도움이 되지 않는 것이다.

Further factors consistently found to add to prediction of performance are integrity and conscientiousness: these do not correlate with IQ23. No additional predictability comes from the number or nature of outside interests; years of education adds little to predictive validity; and the number of courses a person has been on is of no value (so much for how we measure ‘ continuing professional development’). Previous job performance adds to prediction for those already in the profession, but adds nothing at entry. Some of these results are counter-intuitive: this is because IQ overlaps with other things. So a quick learner will have good performance in a previous job which will correlate so highly with IQ that it adds little to predictive validity20.

탈락(academic failure)를 예측하는 요인은 무엇인가?

WHAT FACTORS PREDICT ACADEMIC FAILURE IN MEDICINE?

의학에서 예측인자에 대해 연구한다고 할 때 가장 먼저 집착(?)하는 것은 시험 결과이다. 지금까지 보았을 때 시험 통과의 예측인자를 연구한 논문이 가장 많을 것이다. 이는 중간에 탈락하는 의과대학생에게 들어가는 경제적, 개인적 비용을 고려했을 때 합당한 것이다. 보통 8%~10%정도가 이렇게 탈락한다고 보고되고 있다. 그러나 많은 연구에서 'failure'를 평가할 때 자퇴(exclude)하는 학생 뿐만 아니라 재시험을 보는 학생까지 포함하는 경우가 많고, 따라서 이러한 예측인자들을 주의해서 봐야 한다. 비록 학생들이 고등학교 때에는 0.4%에서 10%사이의 상위권 학생들이었지만, 이 성적과 의과대학 시험 성적에는 상관관계가 있다. 일부 영국 연구들은 일부 과학과목에서 A학점을 받은 것이 의과대학시험 성적을 예측한다고 보고하고 있다. 영국 외 지역에서도 유사한 결과가 있으나 이러한 것이 장기적으로 봤을 때는 성공 또는 실패에 차이를 주지 않는다.

The first thing that strikes anyone exploring the work on predictors in medicine is that we are obsessed with exam results: by far the largest number of papers examines predictors of passing exams. This may be justified because of the economic and personal waste of losing students who begin a medical degree but fail to complete, with loss from schools that select at entry, both in the UK and elsewhere, generally reported between 8% and 10%24, 25, 26, 27. However, most studies assess ‘failure’ in broad terms to include all students who re-take an examination, as well as those who are excluded from the course, so predictors should be treated with caution. Although virtually all students are high academic achievers at school, from the top 0.4%8 to the top 10%29, school and medical exam scores do correlate, with contribution to variability reported between 16%29 and 58%30. Some UK studies show that certain science A levels predict exam success, variously putting biology, chemistry or physics in prime place31, 32, 33, and research from outside the UK reports associations between performance in physical sciences and in medical exams34, 35, 36. Generally this association falls later in the course, with no difference to longer term success or failure37, 38, 39, 40.

비학업적 요인들도 성공 또는 실패를 예측하는 것들이 있는데, 일부 연구자들은 더 나이가 많은 학생일수록 시험에 탈락할 가능성이 높다고 하기도 하나, 다른 연구자들은 이러한 차이는 없다고 보고하고 있다. 몇몇 미국 연구들을 보면 여성 또는 소수인종 학생에서 탈락률이 더 높다고 보고하고 있으며, 한 학교에서는 affirmative action으로 입학한 학생들이 전통적 기준을 통해 들어온 학생들과 졸업하는데 있어서 차이가 없다고 보고하고 있다. 영어가 모국어가 아닌 나라에서는 영어를 얼마나 유창하게 하는가가 중요하며, 미국에서는 소수 인종에서 독해 능력이 학업 성취를 예측해주기도 했다. 비인지적 요인들은 백인 남성보다 여성과 소수 인종에서 더 강력한 예측인자였으며, 여성에 있어서 면접점수와 이전 관련 경험이 시험 점수보다 예측성이 더 높았다. 소수인종 학생에게 있어서는 locus of control과 자기평가 능력이 예측인자였다. 

Non-academic factors also predict exam success or failure. Some researchers report that older students are more likely to fail exams36, 38, 41, but others have not found this42. Several US studies found higher failure rates among women and ethnic minority students, although most eventually graduate36, 38, 41, and one school reported that students admitted through affirmative action were as likely to graduate as those admitted by use of traditional criteria43. Proficiency in English is important for students for whom English is not their first language44, 45, and in the US, reading skills of disadvantaged minority students have been shown to predict academic success46. Non-cognitive factors are stronger predictors for women and ethnic minority students than for white men in the US. For women, interview ratings and previous relevant experience were more predictive than previous exam scores47, while for ethnic minority students, locus of control and ability to self-evaluate were predictors48, 49. One US study showed that different cognitive and non-cognitive factors correlate with academic success in different schools, so different cultures and teaching styles influence outcome50.

미래에 다가올 어떤 failure는 피할 수 없는 것이고, 일부 학생들의 진로희망이 바뀌는 것을 막을 수는 없다. 그러나 두 의과대학에서 학생을 잘 선발하고 잘 지원을 해주면 긍정적 효과를 보여줄 수 있다는 결과가 있다.

It has been argued that we cannot reduce loss further51, because some failure is inevitable and we cannot avoid a few students' wanting to change career. However, two medical schools have shown that careful selection and good support can have a positive impact. 

뉴캐슬 연구 결과를 보면, 낮은 면접점수와 향후 탈락간에는 높은 상관관계가 있었지만, 낮은 학업점수와는 그러한 상관관계가 없었다.

In Newcastle, New South Wales, for five years 50% of students were selected on academic marks alone but underwent a lengthy structured interview which was not used for selection. As a result, some students were admitted with very low interview scores. The remaining 50% were selected from a wider band of academic performance but scored high in interview. Analysis after ten years showed a significant correlation between low interview score and later drop-out but no correlation between academic score at entry and drop-out. Reasons for dropping out were academic failure or a variety of personal reasons, including lack of motivation for study or for medicine28. 

McMaster에서는 remediation을 잘 해준 결과 100명의 학생 중 한 학생만이 학업적 이유로 exclude되었고, 3명은 진로를 바꾸었으며 8%는 remedial이 도움이 되었다.

Another example of low drop-out comes from McMaster University in Ontario, which also invests heavily in selection and in addition offers ‘remediation’ for students having academic difficulty. In one five-year period in a class of 100 students, only one student was excluded because of academic failure, 3 changed careers, while 8% had remedial help52.

우수한 임상 수행능력을 예측하는 것은 무엇인가?

WHAT PREDICTS GOOD CLINICAL PERFORMANCE?

• 임상수행능력은 입학전 학업성적만으로 예측되지 않는다. 
• 나이와 성별 모두 임상 수행능력을 예측하지 못하며, physical science를 과거에 공부했는지도 관계 없다.
• 그러나 이전 영어 학습과 인문학 학습이 임상 수행능력과 상관관계가 있음을 보여준 연구가 있고, 일부 보고서에서는 입학시 면접 점수와의 상관관계를 보여주기도 했다. (입학시 면접과는 상관관계가 없다는 연구도 있다.)
• 지원자에 대해서 매우 자세히 평가하는 학교에서는 공감과 동기부여가 특히 중요하다는 것을 보여주기도 했다.

Investigators looking for early predictors of what makes a good clinician generally use reports from clinical clerkships and from the house officer or intern year. However, we should note that drop-out will mean that some unsatisfactory students will have left before the house officer year. Clinical performance is not generally predicted by pre-entry academic scores1, 35, 53, 54, 55, 56, 57: the one report of correlation between matriculation scores and clinical performance noted that matriculation scores included 50% contribution from school teacher assessment58. Neither age nor gender predicts clinical performance, nor does previous study of physical sciences, but there is evidence that previous study of English and humanities correlates with better clinical performance5, 34, 59. There are some reports of association between clinical performance and admission interviews55, 56, 60, 61, although others reveal no correlation54, 58. In a school that carefully evaluates applicants, empathy and motivation to be a doctor were found particularly important in predicting both clinical and academic success62.

미래 수행능력의 예측인자를 평가하기 위한 가장 타당도 높은 방법은 무엇인가?

WHAT ARE THE MOST RELIABLE PROCEDURES TO ASSESS PREDICTORS OF FUTURE PERFORMANCE?

지금까지 우리가 가지고 있는 최선의 도구는 구조화된 면접이다.

If we can agree that there are certain characteristics that we want to select in prospective doctors, what is the best way of doing this? Research shows that, if we want to add usefully to a measure of intellectual ability in predicting later job performance, our best instrument is the structured interview. 

While an unstructured interview adds about 8% to prediction of subsequent performance, the structured interview adds around 24%63. 

심리검사로 개인적 특성을 알아내는 것이 미래 수행능력 예측에 도움이 될 수 있지만, 만약 선발 도구로 사용하고자 한다면 '정답'을 찾아내기 어렵지 않으므로 이러한 타당도가 손상될 가능성이 있다.

Psychometric tests to measure desirable personal characteristics do predict future performance, but their validity may be compromised if they are used as a selection tool: 

the desired answer is not usually difficult to identify, and applicants who lack integrity are the most likely to manipulate the results64. 

However, some schools have applied psychometric tests at the point of entry rather than using them to select, and have found correlation between these tests and scores given in interview65, 66. This suggests that a well conducted interview may give similar information and that, if constructed to assess desired characteristics such as conscientiousness or helpfulness, it will give a reasonably reliable evaluation20.

이 전에 이 사람을 가르쳤거나 고용했던 사람의 reference가 도움이 될 수 있다. 그러나 법에 따라 employee에 의해서 고소를 당할 가능성이 있고, 평가자의 '동기'가 무엇인지 알 길이 없다.

Character references from a previous employer or tutor have potential to add to prediction. However, legislative changes in the US in the 1980s meant that an employer giving an adverse report could be sued by the employee: as a result, the predictive validity of personal references in the US has fallen to almost zero20. 

The reliability of UCAS references in the UK may be similarly threatened. The motivation of the referee is uncertain: some tutors may feel their first loyalty to their student, others may feel compromised by recent data protection legislation that removes the confidentiality of previous years. 

One medical school in New Zealand has adapted the traditional reference system by writing to head teachers with specific questions, and requesting a rating of the candidate's qualities against the level the head teacher believes to be desirable in a doctor. The long-term predictive validity of this method has not been published, but the school believes it provides valid information and correlates well with other non-cognitive indices (and not at all with academic scores)35. 

일부 학교에서는 small group에서의 수행능력을 통해 실시간으로 대인관계능력을 평가하기도 한다.

Some schools, particularly those which do a lot of small-group work in the course, use an assessment of performance in small groups as a ‘live’ way to assess interpersonal skills29, 52. Evaluation of students in this setting correlates highly with interview scores, and is reported to predict both problem-solving ability and group interaction52.

현재의 '최선의 진료'를 구성하는 것은 무엇인가?

WHAT CONSTITUTES CURRENT BEST PRACTICE?

In summary, the evidence is that we need to select students with good intellectual ability and that examinations, despite limitations, have some validity. For some candidates—e.g. older applicants, or those from disadvantaged social backgrounds—we may want to look for reliable measures of intellectual ability other than the traditional A levels. We seek individuals who are conscientious and have integrity, who are empathic and motivated to become doctors, and who are psychologically robust enough to enjoy a successful medical career. Some medical schools, mainly outside the UK, have already recognized best practice and have put great care and resource into their selection procedures, with well-planned structured interviews, focused reports from schools and evaluation of interpersonal behaviour. As detailed above, there is evidence that this investment is worthwhile in terms of the suitability of students selected, and economically in terms of student loss during the course.

우리는 지금 어디 있는가?

AND WHERE ARE WE IN THE UK?

The greatest single barrier to a more careful selection process in the UK is the amount of resource that each school has to invest. At present, would-be medical students apply to up to four medical schools. All but four of the UK's present twenty-four medical schools interview about 500 to 1000 applicants for their five or six year MB BS courses. Many interviews are still unstructured, and not all schools require their interviewers to be trained. It is unusual for the interview to be more than 15 or 20 minutes, and while brief interviews may be reliable67 the validity of a 15-minute interview is doubtful68. The fact that many candidates are interviewed four times underlines the wastefulness of our present national procedure, but the cost to individual schools to improve radically would be prohibitive. Our present system does not offer society the best practice available: at present we almost certainly turn away people who would make good doctors and accept some who will be mediocre or poor. We could probably reduce loss from the medical course, and so save money and save personal distress among those who were allowed to make an unwise choice. We could also be more just to applicants, and begin the process of education by showing that we are very serious about the kind of personal qualities that we want in a doctor.

The Civil Service, the Armed Forces, and many business corporations have had selection boards for many years: the Civil Service believe these to be money well spent, and industry has gone further and demonstrated their cost effectiveness20. Those medical schools which invest heavily in their selection procedures admit that it is not cheap: on the other hand, it is not cheap to lose students unnecessarily or to employ a poorly motivated or unhappy doctor. There is a strong argument for pooling resources so that applicants get one good assessment instead of four poor ones. This does not preclude medical schools' maintaining individuality and some degree of choice, and candidates will continue to visit schools and attend open days. However, it is time that UK medical schools got together to collaborate in setting up a first-class selection process that is fair to society and fair to all those people who hope to be the doctors of tomorrow.

---

Hughes, P. (2002). Can we improve on how we select medical students?. Journal of the Royal Society of Medicine, 95(1), 18-22.