의과대학에서의 수행능력 예측에 있어서 UMAT점수와 GPA의 비교

의과대학에서의 수행능력 예측에 있어서 UMAT점수와 GPA의 비교

Comparison of UMAT scores and GPA in prediction of performance in medical school: a national study

Phillippa Poole,1 Boaz Shulruf,2 Joy Rudland3 & Tim Wilkinson4

배경

의과대학들은 의학을 공부하고, 훈련받고, 행하기에 가장 적합한 특성을 가진 학생을 선발하기 위한 노력을 지속하고 있다. 인지적 능력을 측정하는 시험과 지난 학업성적 등등을 조합한 방법을 사용하고 있으나 그 예측타당도(predictive validity)는 알려진바가 없다. 이 연구에서는 Undergraduate Medicine and Health Sciences Admission Test (UMAT) 점수, GPA, 혹은 이 둘의 조합의 예측타당도를 비교해보았다.

방법

뉴질랜드에서 UMAT점수로 선발된 2003년 이후의 학생들(1346명)을 대상으로 하였다. 회귀분석 모델에는 인구학적 데이터, UMAT점수, 입학시 GPA와 일상적 평가에서의 수행능력을 포함시켰다.

결과

두 기관간 UMAT에 준 가중치가 다르고, 학생들의 인구학적 구성과 프로그램의 작은 차이들이 있었지만, 결과는 비슷하였다. 입학시 GPA(admission GPA)의 예측력은 2학년, 5학년, 6학년 프로그램에서 가장 뛰어났고, 전체 분산의 17-35%를 설명하였다. 반면 UMAT점수는 분산의 10%이내만을 설명하였다. UMAT점수가 가장 높은 예측력을 보여준 것은 5학년의 필기시험에서 9.9%를 설명한 것이었다. UMAT과 GPA를 합한 것은 대부분의 결과에 걸쳐서 약간의 설명력을 증가시켜줬다. 학년군(grading bands)이나 수가 더 적었음에도 UMAT점수나 GPA모두 인턴 수련을 마친 시점에서의 결과를 예측하지는 못하였다. 

결론

일반적인 인지능력 검사인 UMAT이 의과대학 프로그램에서의 성적을 설명해내는 부분은 GPA의 그것에 비해서 더 적었다. 하지만 UMAT은 GPA와 결합해서 사용할 시에 약간의 설명력을 높여주었다. UMAT점수는 이번 연구에서 다루지 않은 결과를 예측해줄지도 모르기 때문에 추가의 연구가 필요해 보인다.

---

Analyses

Separate analyses were conducted for each programme. Each analysis used regression models to measure the predictive association between UMAT score, admission GPA or combined GPA and UMAT score, and the outcomes listed in Table 1 when background factors were controlled for (age, ethnicity, graduate, rural pathway). For students at the University of Auckland, the interview score was included among the background factors.

An R2 multiple linear regression model was used when the dependent variable consisted of continuous scores; a Nagelkerke pseudo R2 ordinal regression model was used when the dependent variable was categorical (Distinction, Pass, Fail). Seven regression models were established for each outcome in each programme; an example is given in Table 2. One of these models included the background variables alone. The remaining models included background variables with admission GPA, UMAT score, admission GPA and UMAT score combined, or one of the UMAT sections. 

Thus, for each outcome in each year and university, it was possible to quantify the net predictive effect of the UMAT score (overall score or any section score), admission GPA, or both, on outcomes over and above other information available at selection. This was calculated by extracting the percentage of variance explained by the background factors from the total variance explained by the model.

Interactions among the independent variables were not measured because the outcome of interest was the predictive power of these variables. 

Multi-collinearity among the admission GPA and scores on the UMAT sections was measured by the variance inflation factor (VIF). This was < 1.4 for each regression, well below the unacceptable level of 10. There are no published data on the reliability of the UMAT. Although reliability data are calculated for many of the assessment outcomes that contribute to an overall year result, the reliability of the overall result cannot be determined.

---

Med Educ. 2012 Feb;46(2):163-71. doi: 10.1111/j.1365-2923.2011.04078.x.

Comparison of UMAT scores and GPA in prediction of performance in medical school: a national study.

Poole P, Shulruf B, Rudland J, Wilkinson T.

Source

Department of Medicine, Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand. p.poole@auckland.ac.nz

Abstract

CONTEXT:

Medical schools continue to seek robust ways to select students with the greatest aptitude for medical education, training and practice. Tests of general cognition are used in combination with markers of prior academic achievement and other tools, although their predictive validity is unknown. This study compared the predictive validity of the Undergraduate Medicine and Health Sciences Admission Test (UMAT), the admission grade point average (GPA), and a combination of both, on outcomes in all years of two medical programmes.

METHODS:

Subjects were students (n = 1346) selected since 2003 using UMAT scores and attending either of New Zealand's two medical schools. Regression models incorporated demographic data, UMAT scores, admission GPA and performance on routine assessments.

RESULTS:

Despite the different weightings of UMAT used in selection at the two institutions and minor variations in student demographics and programmes, results across institutions were similar. The net predictive power of admission GPA was highest for outcomes in Years 2 and 5 of the 6-year programme, accounting for 17-35% of the variance; UMAT score accounted for < 10%. The highest predictive power of the UMAT score was 9.9% for a Year 5 written examination. Combining UMAT score with admission GPA improved predictive power slightly across all outcomes. NeitherUMAT score nor admission GPA predicted outcomes in the final trainee intern year well, although grading bands for this year were broad and numbers smaller.

CONCLUSIONS:

The ability of the general cognitive test UMAT to predict outcomes in major assessments within medical programmes is relatively minor incomparison with that of the admission GPA, but the UMAT score adds a small amount of predictive power when it is used in combination with theGPA. However, UMAT scores may predict outcomes not studied here, which underscores the need for further validation studies in a range of settings.

© Blackwell Publishing Ltd 2012.

PMID:

 

22239330

 

[PubMed - indexed for MEDLINE]