관찰가능한 행동을 기반으로 프로페셔널리즘 평가하기 : 교훈적 이야기
Basing the Evaluation of Professionalism on Observable Behaviors: A Cautionary Tale
SHIPHRA GINSBURG, GLENN REGEHR, and LORELEI LINGARD
Introduction
전문직종이 속해 있는 기관들은 '전문직업성'을 정의해야 하며, 이것을 어떻게 가르치고 평가할 것인지 그 기준을 개발해야 한다. 최근의 연구에서는 추상적인 정의(이타성, 책무성)에 기반한 것은 조작화하거나 일상 상황에 적용하기 어렵다는 것을 보여준 바 있다. 역량중심평가와 같은 최근의 추세에서는 '관찰가능한 행동'으로 초점이 옮겨가고 있으며 이를 통해서 평가의 신뢰도와 타당도를 향상시키고자 한다. 그러나 이러한 접근법은 두 개의 가정이 중요하다. 하나는 전문직업적 행동과 비전문직업적 행동을 구성하는 것이 무엇인가에 대해서 평가자들이 공통된 기준을 가지고 있어야 한다는 것이며, 다른 하나는 학생들의 행동이 이러한 기준을 충족시키고자 하는 의지 또는 능력을 보여줄 수 있는 지표여야 한다는 것이다.
Professional organizations have worked to define “professionalism” and to develop standards for its teaching and evaluation.1,2 Recent research has suggested that frameworks based on abstracted definitions (e.g., altruism, accountability) may be difficult to operationalize and apply in day-to-day settings, and that behavior-based approaches may be more reflective of the reality of how students and physicians practice.3,4 The current trend towards competency-based assessment has served to shift our focus towards “observable behaviors” as an attempt to improve the reliability and validity of evaluations.5,6 But the legitimacy of these approaches depends on two critical assumptions: that evaluators share a common set of standards for what constitutes professional or unprofessional behavior, and that a student's behavior is an index of his or her willingness or ability to adhere to these standards.
이 두 가지 가정을 점검하기 위해서, 이 연구에서는 가설적 시나리오를 가지고 학생의 행동에 대한 교수들의 인식을 살펴보고, 과연 교수자간, 교수자내 일치성이 있는가를 보았다.
To test these two assumptions, this study used hypothetical scenarios to explore faculty's perceptions about students' behavior and to determine whether there is between- and within-faculty agreement about what medical students should or should not do in professionally challenging situations.
Methods
시나리오
Scenarios
전문직업적으로 대처하기 힘든 상황이 무엇인지에 대한 의대생들의 생각을 연구한 이전 연구를 통해서 다섯 개의 시나리오를 개발하여 녹화하였다. 각 상황은 학생들이 전문직업적 딜레마 상황에서 어떠한 대응을 해야 하는 실제 상황을 극화시켰으며, 각 시나리오는 그 상황에 이어서 학생이 어떠한 행동을 해야 하는 시점에서 종료된다.
Five videotaped scenarios were developed in a previous study, based on medical students' descriptions of professionally challenging situations; these cases were chosen to reflect a range of common, difficult, and realistic dilemmas as perceived by students. Each depicts a dramatization of a real-life event during which a student is placed in a challenging situation that requires action in response to a professional dilemma. Each scenario ends at the point at which the student must act.
비디오 요약
Brief Summaries of Videos
Video 1: A student has just been told by the attending surgeon not to tell a patient the results of a test showing that she has a tumor; in the next scene, the patient asks the student directly what her tests show.
Video 2: A medical resident is trying to get the team out early on the last day of a rotation, but a student wants the resident to review a patient's insulin orders first. The resident says they can wait until Monday, but the student is uncomfortable.
Video 3: A student wants to go watch a bone marrow biopsy, but has just told a patient with dementia that she'd see him right now.
Video 4: A male doctor in a fertility clinic is enthusiastically teaching his students how to examine a male patient's genitals, but the patient has not been asked permission and is uncomfortable. The attending asks the female student to palpate and explain what she feels, but no one is wearing gloves.
Video 5: A student is doing her first thoracentesis while the resident supervises, when a nurse walks in and asks the student if she's ever done one before.
분석
Analysis
데이터는 두 가지 방법으로 분석하였다. 첫째로 학생이 할 것으로 교수들이 예상한 모든 행동에 대한 content analysis를 하였다. 이전 연구에서 학생들에 의해서 만들어진 행동의 리스트로부터 시작하였고, 새로운 행동이 있을 때마다 이를 추가하였다. RA가 모든 행동을 목록화하였고, 이를 두 명의 연구자가 검토하고 차이가 있을 경우 합의를 통해 해결하였다. 데이터베이스는 Excel 스프레드시트에 저장하였다. 추가적으로 이 transcript는 grounded theory에 의해서 분석되었다. 이는 비구조화된 정보의 sampling에 대한 반복적인 작업으로부터 explanatory model을 개발하기 위한 질적연구 방법론의 하나이다. 이 분석의 세세한 결과는 다른 곳에 발표되었으나, 여기서 일부 thematic categories를 강조하기 위하여 사용할 것이다. 요약하면, 우리는 이전의 연구로부터 개발된 템플릿으로부터 시작하여, thematic categories를 개발하고 학생이 어려운 상황에서 특정하게 행동하는 이유에 대한 교수들의 인식을 찾고자 했다.
The data were analyzed in two ways. First, a content analysis was performed to catalogue all of the behaviors that faculty suggested students might do. We began with the list of behaviors generated by students for each video in a previous study, and added new behaviors to the database as they appeared in the transcripts.7 The RA catalogued all the behaviors, which were subsequently reviewed by two of the researchers; discrepancies were resolved by consensus. The database was maintained in an Excel spreadsheet. In addition, transcripts were analyzed by grounded theory, which is a qualitative methodology that is used to develop explanatory models generated by an iterative process of sampling of unstructured data, such as from interviews.9 The results of this analysis will be presented in detail elsewhere, but is included here to highlight some of the thematic categories that arose. Briefly, we began with a template developed in a previous study, and developed thematic categories that captured faculty's perceptions of the reasons that students act in challenging situations.7
결과
Results
Between-Faculty Consistency
The behaviors that faculty most frequently identified for each scenario are shown in Table 1. Based on these results, there appears to be little agreement between faculty regarding what students should do, which suggests that the scenarios were sufficiently challenging—each one provoked many alternatives for action, and there was no single “right” answer prioritized by all respondents. For example, in the fertility clinic video (video 4), four out of 30 faculty thought the students should speak directly to the patient and ensure that he consents to having students examine him, and an additional 16 felt that that would be an acceptable option. However, ten of the faculty did not even mention this as a possibility, despite the fact that on average 5.1 options were mentioned by each respondent for this scenario.
어떤 교수가 '학생이 해야 한다'라고 생각했던 행동이 다른 교수에게는 '학생이 하지 말아야 할 행동'으로 여겨지기도 했음
Further, in several cases, what some respondents thought students should do was exactly what others felt students should not do. For example, in video 2, six faculty felt that the student should pursue the matter of the insulin orders directly with the staff physician (i.e., go over the resident's head), while two said a student should not do that; five thought the best thing to do would be to push the same resident further to get the orders written, while two thought this was a bad idea; and although 15 of the faculty said that students should not simply obey the resident and leave with the team for the weekend, three felt that this was what a student should do.
이타심과 같은 추상적인 원칙의 적용에 있어서도 교수간 차이가 나타났음
Discrepancies between faculty also appeared as inconsistencies in the application of abstract principles, such as altruism. For instance, in video 3, the student must choose between spending the necessary time with a patient or going to watch a bone marrow biopsy. Seven faculty felt that the student should go see the procedure first, because she may not get another opportunity, while two thought she should forego the bone marrow and spend whatever time is necessary with the patient. As one respondent (M10) put it, “ … it's a conflict of interest in that the student is asked to choose between something that furthers her interest … go get educated … versus something that's in the patient's interest, which is to hear the truth again…” This attending feels the best option for the student is to put the patient first. However, a different respondent (M13) thought otherwise: “… you owe it to yourself and ultimately to your patients … to take every educational opportunity … it is something they need to learn to help future patients.”
Within-Faculty Consistency
개별 응답에 대해서 두 가지 측면을 보았다. 하나는 대안으로서 제시한 것들의 숫자이며 다른 하나는 여러 시나리오에 걸쳐서 어떤 원칙을 적용하는가였다. 모든 시나리오에서 교수들은 두 개 이상의 대안을 제시하였는데, 교수에 따라서 일부 시나리오에는 2개를 다른 시나리오에는 9개의 대안을 제시한 교수도 있었다.
To determine whether differences in opinion were due to stable individual differences in faculty respondents, we explored two aspects of individuals' responses: the number of alternatives suggested, and the application of principles across scenarios. As Table 1 shows, for each video there was at least one respondent that generated only two alternatives for action; however, this was not the same individual in each scenario. For example, S14 gave only two suggestions in video 1 but nine options in video 2, whereas S5 gave five options in video 1 and two in video 2. Therefore, it was not the case that some faculty were more “thoughtful” than others, or that some found all of the scenarios “easy” while others struggled. Rather, the relative complexity of individuals' responses varied idiosyncratically across scenarios.
또 다른 발견은 개별 교수들이 여러 시나리오에 대해서 대안을 제시할 때, 추상적인 원칙의 적용에 있어서 내적 일관성이 부족했다는 점이다.
A second key finding was that individual faculty often suggested alternatives across videos that were not internally consistent in their application of abstracted principles. For example, in video 5, one faculty (S7) responded that the student should tell the patient that it's her first time doing a thoracentesis, and specifically stated that “You can't be evasive here … you should never lie.” However, this same respondent in video 4 said the students should tell the attending that they forgot that they had a seminar to attend downtown, as a means of extricating themselves from an uncomfortable situation. For S7, then, the principle of honesty was not of overriding importance across cases—it was in one scenario, but not in another. Differences in context may explain this inconsistency: in video 5 the issue was whether or not to be honest with a patient, while in video 4 the dishonesty (making up an excuse to leave) did not involve the patient at all.
그러나 '정직성'을 적용하는 것에 대한 비일관성은 같은 시나리오 내에서도 있었다.
However, inconsistency in the application of “honesty” was also seen within the same scenario. In video 1, M10 stated that “What [the student] absolutely should not do, is say, ‘Everything is okay,' because that's actually lying. Because everything is not okay.” This respondent further stated that lying is “morally wrong” and is never the right thing to do. However, later in the interview he went on to say that “the student could say that ‘With regards to the surgery everything is okay …' I could see her giving an answer like that and I don't think that would be absolutely wrong.” To M10, this appears to be an acceptable response for the student because this withholding of information directly relevant to the patient's question is not “actually lying, or misrepresenting the truth.”
고찰
Discussion
의과대학 학생에 대해서 어떤 것이 직업전문적 행동이고 어떤 것이 그렇지 않은지에 대한 교수들의 생각은 교수간, 그리고 교수 내에서도 disagreement가 있었다. 이러한 개념 자체가 완전히 새로운 것은 아니며 프로페녀널리즘의 추상적 원칙이 적용될 때의 비일관성을 보여준 연구가 있었기 때문이다. 그러나 이전 연구들은 자신이 어떻게 할 것인가에 대한 것이었지 다른 사람들이 어떻게 해야 하는가에 대한 연구가 아니었다. 의과대학생들의 행동에 대해서 '이상적인'것 또는 적절한 것이 무엇이냐고 물었을 때, 교수들의 대답은 원칙을 적용한 이상적이고 표준화된 답일 것이라는 생각을 하기 쉽다. 그러나 - 심지어 이타심 조차 - 교수들은 학생의 행동에 대해서 쉽게 operationalize하지 못했다. 이러한 응답은 원칙의 '정의'와 '적용' 모두에 대한 유연성을 보여준다. 추상적인 이상을 방어가능한 행동으로 적용시키는 것은 일반적으로 생각하는 것 보다 훨씬 복잡하다.
These results suggest that there is substantial disagreement both between and within faculty about what constitutes professional and unprofessional behavior in medical students. This concept in itself is not entirely new, in that other studies have shown inconsistencies in how certain abstract principles of professionalism are applied. For example, physicians and residents have been shown to be more willing to use deception in some scenarios than others, and disagreements have been shown amongst students about what does or does not constitute cheating.10–12 But these studies have focused on what individuals themselves might do, not what they felt others should do. One might postulate that when asked for an “ideal,” or at least appropriate, solution for medical students in common professional dilemmas, the responses from faculty would be more idealistic and standardized, based on the application of the principles espoused by the profession. However, even altruism, which is often considered the highest of ideals, is not so easily operationalized by faculty when considering students' behavior—in some cases, putting the student's patient first was not the preferred option. Such responses demonstrate a flexibility in both the definition of principles (such as honesty or altruism) and their application. The translation of abstracted ideals into defensible behaviors is therefore more complex than is usually acknowledged.
추상적인 이상과 원칙이 행동으로 잘 적용되지 못한다는 사실은 중요한데, 이러한 사실은 이미 인지된 바 있다. 그러나 우리의 결과가 제시하는 정말 심각한 것은 그 행동이라는 것 조차 '프로페셔널리즘'의 지표가 아닐 수 있다는 것이다. (교수에 따라 생각하는 것이 다르기 때문에)
The fact that abstracted ideals and principles do not map easily onto behaviors is important, and has previously been recognized.4 But the really critical issue our results suggest is that behaviors themselves may not in fact be obvious or transparent indications of “professionalism.”
Consider video 1, in which a student has been told not to disclose a specific test result (which indicates a tumor) to a patient, who then asks the student, “What do my tests show?”
To one faculty member, the dilemma involves a values conflict between two options: obey the surgeon and lie to the patient, or disobey the surgeon and disclose the diagnosis in order to be honest with the patient. As M5 put it, “The student has been asked a direct question by the patient. The student knows the result… Essentially, to not tell the truth about the test result, would, in my mind be incorrect.” From the point of view of this faculty, lying to the patient is seen as unprofessional.
However, for another faculty member, the values conflict might be construed differently: tell the truth about the test result and risk distressing the patient (because you're not equipped to answer all of her inevitable questions), or lie to the patient to spare her feelings, at least for the time being, as S16 stated: “They shouldn't give the patient information … that they can't adequately discuss … and giving the patient half the information is worse than not giving the patient any information at all, as far as I'm concerned.” From the point of view of this faculty, lying to the patient may be the more professional thing to do.
이러한 경우에 있어서 '정직성'이라는 원칙은 어느 상황에서도 갈등의 요소가 된다. 프로페셔널리즘을 평가하는데 있어서 우리의 관점은 '태도'에서 '관찰 가능한 행동'으로 바뀌어왔다. 그러나 우리나 만약 '행동'에만 초점을 두고 맥락을 고려하지 못한다면 attribution error를 저지르는 것이다. 우리가 가진 현재의 평가 체계는 특정 행동을 해석하는데 있어서 맥락이 불가피하게 영향을 줄 수 밖에 없다는 것을 무시하는 것이며, 특히 행동의 동기에 대한 해석을 무시하는 것이다.
In both of these values conflicts, the principle of “honesty” is at stake—but if our first faculty member had witnessed the student lying to the patient, s/he might rate that student (or at least her behavior) as being unprofessional, whereas if the second had been present the student might be rated as being very professional, not despite the lie but because of it. In terms of evaluating professionalism, our focus has shifted away from “attitudes” and towards observable behaviors, as a necessary step towards improving reliability and validity. However, if we focus only on behaviors, and fail to take the context into account, we risk making what is known as the fundamental attribution error; that is, the tendency to underestimate the situation and overestimate the extent to which a behavior reflects stable traits or attitudes in an individual.13,14 Our current evaluation systems tend to ignore the fact that context inevitably drives an observer's interpretation of behaviors, and critically, the interpretation of the motivations for the behaviors.3
우리가 말하고자 하는 것은 '행동'이 아무런 관련이 없다는 것은 아니다. 분명히, 학생이 어떤 행동을 하는가는 중요하다. 그러나 프로페셔널리즘을 정밀하게 평가하기 위해서는 추가적인 차원이 필요하며, 이는 행동 그 자체로는 정확한 판단을 내리기에 충분한 정보를 제공하지 못하기 때문이다. 어떻가 한 학생이 특정한 딜레마를 해결하는지, 그 상황에서 그 학생의 어떤 가치가 갈등을 일으키는지가 중요한 정보이다. 또한 학생의 행동이 있게 만든 논리적 근거가 나중에 이 학생이 전문직으로서 딜레마 상황을 마주했을 때 어떻게 의사결정을 내릴 것인가에 대한 중요한 통찰력을 제공할 것이다. 마지막으로 이러한 결과는 오랜 시간에 걸친 다양한 관찰에 근거해서 평가해야 함을 강조한다.
We are not suggesting that behaviors are irrelevant—clearly, what a student does is still important. However, sophisticated evaluation of professionalism requires an additional dimension, as behaviors alone do not give us all of the information we need to make accurate judgments. Knowing how a student construes a particular professional dilemma, and what values s/he perceives as conflicting, is critical information.3 Furthermore, analyzing the reasoning behind students' behaviors may give us significant insights into how they make decisions when faced with professional dilemmas.7 Finally, these results reinforce the need for evaluations to be based on multiple observations over time, rather than on a single instance of behavior.
이 결과는 '프로페셔널리즘'의 기준에 대해서 교수들에게 일관되고 명백하게 공유되는 어떤 것은 없음을 보여준다.
Our results suggest that there is no apparent shared standard of “professionalism” that is applied uniformly and consistently by faculty. Although this study was performed in a single institution, it involved faculty from two large clinical departments; however, it is unclear whether faculty from different types of institutions (e.g., academic vs. community) or clinical disciplines (e.g., psychiatry or pediatrics) would show similar patterns of responses. Nonetheless, we would suggest that the translation of abstracted principles into action will always be complex, and depends on the context and values conflict inherent in each situation. Moreover, methods of evaluating professionalism, while retaining some focus on what students actually do, should look beyond observable behaviors to include the reasoning behind them, in order to develop a more accurate assessment of a student's developing professionalism.
Basing the evaluation of professionalism on observable behaviors: a cautionary tale.
Abstract
PROBLEM STATEMENT AND BACKGROUND:
METHOD:
RESULTS:
CONCLUSIONS:
'Articles (Medical Education) > 전문직업성(Professionalism)' 카테고리의 다른 글
| ○의학전문직업성의 개념과 발전 방향(KMER) (0) | 2014.11.26 |
|---|---|
| 의사소통역량 평가하기 : 기존 도구들에 대한 평가 (0) | 2014.11.06 |
| 문화적 의사소통 역량의 핵심요소 (0) | 2014.10.27 |
| 독일어권 국가 의학교육에서의 의사소통과 사회적 역량 : The Basel Consensus Statement (0) | 2014.10.20 |
| 프로페셔널리즘 평가: the Ottawa 2010 Conference 제안 (0) | 2014.09.25 |