의사들의 자기평가의 정확성 : 역량에 대한 외부 평가와의 비교

(출처 : http://visual.ly/self-assessment-2011)

자기평가(self assessment)와 자기주도학습은 의료 전문직의 핵심이라고 할 수 있으며, 이는 환자에게 행해지는 의료의 질과 곧바로 연결된다. 미국의 의사들은 대부분의 전공과별 위원회에서 지침(rubric)으로 정해놓은 것에 따라 CME에 참여함하여 평생학습을 해야 하고, CME학점을 이수해야 한다. 

SELF-ASSESSMENT AND SELF-directed, lifelong learning have long been mainstays of the medical profession—they are activities presumed to be linked closely to the quality of care provided to patients.1 Physicians in the United States must demonstrate their engagement in lifelong learning by choosing and participating in continuing medical education (CME) activities2 and acquiring CME credit, which is mandated by the majority of state medical boards under the rubric of states’ medical practice acts.3

American Board of Medical Specialties에서는 자기평가와 평생학습(lifelong learning)을 자격증을 유지하기 위한 4가지 요소 중 하나로서 명시하고 있다. 더 나아가 미국 내과 위원회(American Board of Internal Medicine)에서는 자격의 갱신(recertify)를 위해서 의사 스스로 자기자신에 대한 평가를 할 수 있는 능력을 필요로 하는 지속적 전문성 개발(continuous professional development)를 요구하고 있다. 또한 미국, 캐나다, 그리고 다른 여러 나라에서 졸업후교육(GME)의 기본은 진료기반학습(practice based learning)과 자기평가를 기반으로 한 향상이다.

Self-assessment and lifelong learning were adopted by the American Board of Medical Specialties explicitly as 1 of 4 elements in its Maintenance of Certification program.5 Furthermore, diplomates of the American Board of Internal Medicine who choose to recertify recertification, a process focused on continuous professional development that requires the capacity of physicians to self-assess.6 In graduate medical education, the issue of practice-based learning and improvement based on self-assessment is a central tenet of professional development in Canada,7 the United States,8 and in other countries. 9

이 과정에 들어있는 요소들은 의사들이 스스로의 학습에 대한 필요를 판단하고 그에 맞는 학습자료를 찾가알 수 있는 능력에 달려 있다. 

Each of the elements in this chain—the emphasis on self-assessment, self-directed lifelong learning, the acquisition of CME credits and their use for medical relicensure, accreditation, and ongoing certification—is heavily dependent on the ability of physicians to determine their own learning needs and find resources to meet them

the emphasis on self-assessment, 

=> self-directed lifelong learning, 

=> the acquisition of CME credits and their use for medical relicensure, 

=> accreditation, 

=> and ongoing certification

자기평가라는 단어가 여러가지 종류의 활동을 포함하고 있지만, 이 논문에서는 자가 관리 시험보다는, 자기측정(self-rating, self-audit)과 임상 수행 능력에 대해 보고자 한다.

While the term self-assessment is used to describe many types of activities, we were interested in considering the aspects of “self-rating” or “self-audit” in contrast to the use of self-administered examination of knowledge or clinical

performance.

방법

Data Selection

Studies that focused on a comparison between physicians’ self-assessments as determined by self-ratings and 1 or more external measures of related competencies were included.

Data Extraction

The following information was extracted from each article: study population; content area and self-assessment domain of the study; methods used to measure the self- assessment of study participants; methods used to observe or measure participants’ competence or performance; existence and use of quantifiable measures; and study outcomes.

결과

Domains of Self-assessment

Methods of Self-assessment

Methods of External Assessment

Methods of Comparing Self- and External Assessments

Accuracy of Self-assessment

20개 논문의 자기평가와 외부평가를 비교한 결과, 13개의 논문에서 관련이 없거나, 오히려 음의 상관관계를 가지는 것으로 나타났다.

Of the 20 comparisons between self and external assessment, 13 demonstrated little, no, or an inverse relationship between self-assessment measures and other indicators.12-18,20,21,23,26,27

전반적으로 연구 결과는 아주 작거나, 없거나, 심지어 음의 상관관계를 보였으며 이러한 경향은 훈련의 수준, 경험의 차이, 연구의 연도에 관계없었다. 반대로 7개의 논문에서는 양의 상관관계를 보였다.

Overall, the proportion of studies reporting little, no, or inverse relationships did not appear to vary by level of training or experience or by year of study. In contrast, 7 comparisons12,15,19,22,24,25,28 demonstrated positive associations between self-assessment and most external observations

COMMENT

Relationship Between Self-rated Assessment and External Assessments

이 systematic review에서 살펴본 대부분의 연구에서, 의사들은 자기에 대한 평가를 그다지 정확하게 하지 못하는 것으로 나타났다. 

This systematic review found that in a majority of the relevant studies, physicians do not appear to accurately self assess. Weak or no associations between physicians’ self-rated assessments and external assessments were observed.

이러한 결과는 훈련의 수준, 전공, 자기평가 영역, 비교 방법에 상관없이 나타났다. 하지만 이들 결과는 전혀 새로운 것이 아닌데 Sibley 등은 비슷한 결과를 20년 전에 발표한 바 있고, Gordon과 Dunning도 비슷한 결과를 발표한 바 있다. 이러한 연구결과는 다른 분야에서의 연구결과와도 일치한다. 예컨대 법학, 공학, 행동과학, 심리학 등의 학문에서 Falchikov와 Boud는 자기 평가와 외부 평가의 연관성은 평균적으로 0.39 정도라고 밝혔다.

In the studies indicating poor or limited accuracy of selfassessment, this finding was independent of level of training, specialty, the domain of self-assessment, or manner of comparison. These findings are not new. Sibley et al29 reported similar findings more than 2 decades ago, as did subsequent studies by Gordon30 and Dunning et al.31 The findings are consistent with studies in other disciplines. For example, in a meta-analysis of quantitative self-assessment studies in law, engineering, guidance counseling, behavioral science, psychology, and medicine, Falchikov and Boud32 noted correlations between self- and external assessments of student performance ranging from 0.05 to 0.82, with a mean of 0.39. 

보건의료직으로 한정지으면, Gordon은 그 상관성이 0.02~0.65 정도라고 밝혔다. 

Within the health profession, Gordon30 found that correlations for self-assessments of knowledge ranged from 0.02 to 0.65. Furthermore, despite our finding in 2 studies24,28 that specific selfassessment may be reliable predictors of performance, Eva et al33 found that poor correlations persist even when domains are well-defined. 

마지막으로, 이러한 결과들로부터 가장 눈에 띄는 것은, 외부 평가에서 부정적 평가를 받은 사람이 자기 평가도 부정적으로 내렸다는 것이다.

Finally, perhaps of greatest concern are the findings that those who perform the least well by external assessment also self-assess less well.

Construct and Study of Self-rated Assessment

첫째로, 자기측정평가는 연구가 용이한 분야가 아닌데, 이는 주로 그 특성이 완전히 개발되거나 테스트되지 않았기 때문이다.

First, the construct of self-rated assessment itself is not easily studied, in large part because its nature is neither fully developed nor tested.10

두 번째로 자기 평가 연구를 수행할 경우, 연구자들은 다음의 과정을 통해서 연구의 rigor와 reportability를 높여야 한다.

Second, if such studies of self-assessment are undertaken, researchers should augment study rigor and reportability by

better describing their populations, sampling frame, and methods; 

more clearly differentiating between types of self-assessments; 

attempting to resolve questions of volunteer bias; 

and articulating best practice templates for studying and reporting self-assessment compared with external assessment

Limitations

문헌조사를 저자 중 한 사람과 외부의 전문가가 같이 수행했음에도 의학 관련 문헌에 대한 제목 정보의 부족으로 일부 연구들은 누락되었을 수 잇다.

First, while literature searches were conducted by one of the authors and an independent information specialist to provide an exhaustive coverage of the literature, the lack of extensive Medical Subject Headings in the literature databases could have contributed to not retrieving some studies

일부 연구는 방법, 결과, 통계법을 잘 밝히지 않아서 이를 통해서 가설에 대한 근거를 세우고, 일반화하기에 한계가 있었다.

Second, some studies lacked full descriptions of methods, outcomes, and use of statistical tests, limiting our ability to describe the studies more fully, to develop explanatory hypotheses, or to generalize. Third, the domains of competence or performance in these studies, the tools used to measure them, and the assumptions (such as predictions of future performance on a test, or self-ratings in past performances as a teacher) were variable, precluding a meta-analytic approach.

리뷰한 논문의 숫자가 많지 않다고 할 수 있다.

Finally, it can be argued that the relatively small number of studies found in this review

Assessment Formats and Content

만약 의사들이 이러한 분야에서 잘 해내지 못하고 있다면, 자기 평가에 있어서 새로운 initiative와 형식이 도입될 필요가 있다.

If it is true that physicians perform poorly in this domain, new initiatives and formats are needed to assist the self-assessment process and to more accurately promote and assess broader domains of competence such as professionalism and lifelong learning.38

첫번째로, 더 포괄적인, 지속적 전문성 향상 과정에 대한 향상을 측정할 수 있어야 한다.

First, such measures might include the development of a more holistic continuing professional development process involving learning portfolios, documenting practice-based learning and improvement activities, creating less general and more detailed learning and practice objectives, and addressing the general competencies espoused by the Accreditation Council for Graduate Medical Education.8 

두 번째로 훈련을 통해서 자기평가와 외부 평가의 차이가 줄어들 수 있다.

Second, training may reduce the variation between self- and external assessments by encouraging the internalization of objective measurements or benchmarks of performance.10

세 번째로 자기평가 방식의 향상이 근본적으로 어렵다고 하면, 다른 피드백 방법이 필요할 수 있다.

Third, given that some improvement needs (eg, those in the psychosocial realm)13,45,46 may be more difficult to self-assess, methods such as multisource feedback (360°) evaluations may be a necessary next step, particularly when interpersonal skills, communication skills, or professionalism needs to be evaluated.47 

넷째로, 면허나, 면허갱신, 환자안전 등의 이슈를 고려할 때 객관적인 측정기준을 위한 많은 고려가 필요하다.

Fourth, objective measures of competence and performance deserve serious consideration, especially when issues of medical licensure, recertification, quality, and patient safety are paramount.

마지막으로, 각 전공과에서는 정기적으로 구성원들에게 근거중심의 학습목표를 제공하는 역할을 할 수 있어야 한다.

Finally, specialty societies and others can increase their role in providing current evidence-based learning objectives on a regular basis to members of their discipline, giving external markers of competence.

---

JAMA. 2006 Sep 6;296(9):1094-102.

Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.

Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L.

Source

Knowledge Translation Program of the Li Ka Shing Knowledge Institute at St Michael's Hospital, University of Toronto, Toronto, Ontario, Canada.

Abstract

CONTEXT:

Core physician activities of lifelong learning, continuing medical education credit, relicensure, specialty recertification, and clinical competence are linked to the abilities of physicians to assess their own learning needs and choose educational activities that meet these needs.

OBJECTIVE:

To determine how accurately physicians self-assess compared with external observations of their competence.

DATA SOURCES:

The electronic databases MEDLINE (1966-July 2006), EMBASE (1980-July 2006), CINAHL (1982-July 2006), PsycINFO (1967-July 2006), the Research and Development Resource Base in CME (1978-July 2006), and proprietary search engines were searched using terms related to self-directed learning, self-assessment, and self-reflection.

STUDY SELECTION:

Studies were included if they compared physicians' self-rated assessments with external observations, used quantifiable and replicable measures, included a study population of at least 50% practicing physicians, residents, or similar health professionals, and were conducted in the United Kingdom, Canada, United States, Australia, or New Zealand. Studies were excluded if they were comparisons of self-reports, studies of medical students, assessed physician beliefs about patient status, described the development of self-assessment measures, or were self-assessment programs of specialty societies. Studies conducted in the context of an educational or quality improvement intervention were included only if comparative data were obtained before the intervention.

DATA EXTRACTION:

Study population, content area and self-assessment domain of the study, methods used to measure the self-assessment of study participants and those used to measure their competence or performance, existence and use of statistical tests, study outcomes, and explanatory comparative data were extracted.

DATA SYNTHESIS:

The search yielded 725 articles, of which 17 met all inclusion criteria. The studies included a wide range of domains, comparisons, measures, and methodological rigor. Of the 20 comparisons between self- and external assessment, 13 demonstrated little, no, or an inverse relationship and 7 demonstrated positive associations. A number of studies found the worst accuracy in self-assessment among physicians who were the least skilled and those who were the most confident. These results are consistent with those found in other professions.

CONCLUSIONS:

While suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment.

Comment in

• Self-assessment in lifelong learning and improving performance in practice: physician know thyself. [JAMA. 2006]