의학교육에서 "성과 연구(outcomes research)"의 초점에 대한 재고려

Perspective: Reconsidering the Focus on “Outcomes Research” in Medical Education: A Cautionary Note

David A. Cook, MD, MHPE, and Colin P. West, MD, PhD

Abstract

Researchers in medical education have been placing increased emphasis on “outcomes research,” or the observable impact of educational interventions on patient care. However, although patient outcomes are obviously important, they should not be the sole focus of attention in medical education research. The purpose of this perspective is both to highlight the limitations of outcomes research in medical education and to offer suggestions to facilitate a proper balance between learner-centered and patient-centered assessments. 

The authors cite five challenges to research using patient outcomes in medical education, namely 

(1) dilution (the progressively attenuated impact of education as filtered through other health care providers and systems), 

(2) inadequate sample size, 

(3) failure to establish a causal link, 

(4) potentially biased outcome selection, and 

(5) teaching to the test. 

Additionally, nonpatient outcomes continue to hold value, particularly in theory-building research and in the evaluation of program implementation. 

To educators selecting outcomes and instruments in medical education research, the authors offer suggestions including to clarify the study objective and conceptual framework before selecting outcomes, and to consider the development and use of behavioral and other intermediary outcomes. Deliberately weighing the available options will facilitate informed choices during the design of research that, in turn, informs the art and science of medical education.

---

지난 십년간 의학교육 학계는 "성과 연구"에 상당히 몰두해왔다. '성과'라는 단어가 '결과로서 따라오는 것'을 의미함에도 '성과'를 논의할 때 이 단어는 흔히 '임상 결과(clinical outcome)'을 의미하곤 한다. 즉, 어떤 개입의 효과가 환자나, 환자를 돌보는 의사의 행동에 영향을 줘야 한다는 것이다.

Over the past decade, the academic community has seen increased emphasis on “outcomes research” in medical education.1–3 Although the word outcome refers generally to “something that follows as a result or consequence,”4 in the discourse of outcomes research in medical education, this word refers to clinical outcomes—that is, an intervention’s impact on patients and sometimes on physician behaviors during patient care.2

연구자들은 환자 결과(patient outcomes)를 프로페셔널한 행동에 대한 연구라든가, 의사들의 의사소통, 외과에서의 수련 등등의 성과연구에 활용해왔다. 졸업후교육에서 종적(longitudinal)으로 환자결과를 수집해온 연구도 있다.

Investigators have used patient outcomes in studies of professional behavior,11 physician communication,12 surgical training,13 and continuing medical education.14 Others have reported systems to collect patient outcomes longitudinally in postgraduate education.15,16

환자결과가 중요하긴 하나, 환자에 미치는 영향이나 의사의 행동은 나타날 수 있는 '성과'의 일부분일 뿐이다. 50년 전 Kirkpatrick은 4단계 모델을 제시하여 널리 인정받았다. 이 4단계에 의하면 환자결과는 강조할 가치는 있으나 의학교육에서 유심히 봐야 할 유일한 것은 아니다.

Although the patient outcomes research movement is important, patient effects and physician behaviors make up only a subset of possible outcomes. Fifty years ago, Kirkpatrick17 proposed a widely accepted, four-level model of training program outcomes, comprising, first, reaction (satisfaction); followed by learning (knowledge, skills, and attitudes); then, behaviors in practice; and, finally, results (effects on the object of interest, such as in medicine, patients). Patient outcomes warrant emphasis, but they should not constitute the sole focus of attention in medical education.

환자결과에 과도하게 집중하는 것은 오히려 역설적으로 의학교육의 art and science에 대해 혼란스럽게 한다. 이는 임상연구에서 mortality에만 관심을 갖는 것과 유사하며, 환자와 관련된 다른 것들(QOL)을 생각하지 않게 만들고, 연구질문의 범위를 제한하며, 많은 연구를 불가능하게 만들어버린다.

An excessive emphasis on patient outcomes may paradoxically distract investigators from advancing the art and science of medical education overall. Such an emphasis on patient outcomes in medical education would be akin to focusing clinical research outcomes on mortality, which would neglect other outcomes important to patients (such as quality of life), restrict the type of research questions asked (not all interventions are designed to prolong life), and make many studies infeasible (e.g., randomized trials with mortality outcomes typically require long periods of follow-up and incur high expense).

환자결과를 의학교육연구에서 활용할 때 일어날 수 있는 몇 가지 한계를 제시하고자 한다.

We recognize several drawbacks to using patient outcomes in medical education research.

Challenges and Limitations of Research Using Patient Outcomes in Medical Education

희석 

Dilution

의사가 '하는 것(behavior)'과 환자결과(result)간의 관계는 간접적이다.

The link between what a physician does (Kirkpatrick’s behaviors) and what patient outcomes reflect (Kirkpatrick’s results) is indirect

또한 추가적인 요인들이 영향을 줄 수도 있다. (다른 팀 동료, 정책, 보험 등)

Moreover, in most instances, additional factors—such as other members of the health care team (nurses, pharmacists, trainees, etc.), institutional policies, and insurance plan requirements—also have an effect.

궁극적으로 이들 교란요인은 의사의 행동을 '희석'시켜서 교육의 효과로 인해 나타나는 활동의 결과를 없는 것처럼 보이게 한다.

Ultimately, these confounding factors or conditions effectively dilute the physician’s actions19 and diminish the observed effect of the educational activities that preceded those actions. 

연구자들에게는 두 가지 옵션이 있다. 개입의 영향을 더 키우거나, 작은 효과조차 잘 나타나도록 샘플 크기를 키우는 것이다. 그러나 이 둘 모두 교육분야 연구에서는 이상적이지 않다.

Researchers have two options: to increase the initial impact of the intervention (so that even after dilution the effect remains strong) or to enroll a sample size large enough to detect even small (dilute) effects. Neither solution is ideal in education research.

초강력한 개입법을 만드는 것은 매력적으로 보이지만, 우리는 이미 이러한 개입법을 달성하려면 다방면의 접근이 필요함을 알고 있다.

Creating an exceptionally strong intervention sounds attractive at first.However, we have observed that strong interventions nearly always require a multifaceted approach to training,drawing on multiple instructional 

modalities (e.g., combinations of textbook, video, lecture, small groups, computer-assisted instruction, standardized patients, other simulation, and clinical encounters) and instructional methods (practice cases, group discussion, self-assessment questions, feedback, mastery learning, etc.).

또한 강력한 개입은 약한 개입 또는 무개입과 비교했을 때 효과가 크게 나타난다. 연구자들의 지적과 같이 'comparative effectiveness research'가 점차 중요해질 것이고, 이러한 연구에서 '효과크기'는 오히려 작은 경우도 흔하다.

Moreover, strong interventions often show a large effect only when judged against a weak comparison intervention or no intervention. As researchers seek to advance the science of education, the importance of comparative effectiveness research (side-by-side comparisons of two or more active educational interventions) will increase.21 The expected effect size in such research is often rather small.22–24

일부 연구자들은 초점을 진료를 하는 의사집단에 맞춰야 한다고 말한다. 예를 들면 훈련 프로그램이나 큰 코호트에 대한 정보를 모아서 보아야 한다는 것이다. 그러나 이렇게 하면 의사 개개인의 중요성이 낮아질 수 있다.

Some have argued that the community might shift its attention to assessing practitioner groups, such as looking at the aggregate data for a training program15 or large cohort.25 Although examining aggregate group data makes sense at a programmatic level (i.e., for identifying curricular priorities and gaps, or for demonstrating the overall effectiveness of a program), doing so minimizes the importance of individual providers

실헌가능성, 샘플 크기

Feasibility: Sample size

의학교육연구에서 활용가능한 샘플의 크기로 인해 연구의 효과를 적절히 보여주지 못할 수 있다.

In medical education research, the conveniently available sample size (e.g., the number of participants in a training program) is often inadequate to appropriately power the study.

교육적 개입의 영향력이 희석될 수 있기 때문에, 측정도구역시 완벽하지 않기 때문에 이런 연구는 큰 샘플 사이즈나, 큰 영향력(큰 효과크기)가 있는 개입을 필요로 한다. 그렇지 않다면 통계적으로 유의미하지 않은 결과가 나올 것이다.

Because the effect of the educational intervention— even if successful—would be diluted (as above), and the measurements would be imperfect, such a study would either need a very large sample or an intervention with a huge impact (large effect size). Anything less would likely result in nonstatistically significant findings.

일부 연구자들은 더 많은 환자를 enroll하여 학습자 샘플이 부족한 것을 극복하고자 했으나, 결과를 분석할 때 'clustering'을 잘 설명해야 하며, clustering은 effective sample size를 낮춘다. 그러나 유감스럽게도 연구자들은 종종 clustering을 잘 조절하지 못하여 잘못된 분석이나 이상한 분석을 하곤 한다.

Some investigators have attempted to overcome the barrier of an insufficient learner sample by enrolling more patients. However, they must then account for clustering when analyzing the data, and clustering lowers the effective sample size.30 Regrettably, researchers often fail to adjust for clustering (as documented in clinical research),31 resulting in flawed analyses and questionable interpretations, as noted in recent systematic reviews in education.8,32

다양한 프로그램에서 학습자를 enroll하여 이러한 한계를 극복하고자 하는 연구자들도 있고 이것이 성공하기도 하지만, 프로그램들 사이에 개입과 성과 측정이 동일한지를 확인해야 한다.

Other researchers have attempted to increase sample size by enrolling learners from multiple programs—either different training programs within an institution or similar programs from different institutions. Although doing so often leads to success, many important research questions do not lend themselves to multiprogram study—especially questions that require interventions and outcome measures to be implemented equally across programs.33

인과관계 성립 실패

Failure to establish a causal link

교육자들은 study rigor를 향상시키기 위하여 환자 결과에 집중하고는 한다.

Educators might expect a focus on patient outcomes to improve study rigor.

많은 의학교육연구들이 'nonrandomized' design을 사용하는데, 이것들은 randomized 연구보다 인과관계가 훨씬 약하다.

Many medical education studies using patient outcomes employ nonrandomized designs such as concurrent cohort designs, retrospective designs with historical controls, and single-group cross-sectional designs. Such designs allow much weaker causal interpretations than do randomized studies.34

인과관계가 약해지는 또 다른 이유는 교란변수인데, 다면적 개입방법의 활용 또는 multiple instructional feature가 다양한 연구간 비교하는 것 등이 흔한 원인이다. 또 다른 원인은 systematic variation이며, 무작위연구조차 교란변수를 완전히 통제하지는 못한다.

Another source of weakened causal inferences is confounding, which occurs when the observed effect can plausibly be ascribed to a known or suspected influence other than the object under study. Multifaceted interventions (as described above in the Dilution section) and comparisons in which multiple instructional features vary simultaneously represent two common sources of confounding in education research.35 A third source of confounding, particularly problematic for patient outcomes research, is systematic variation in patient populations across physicians (e.g., some physicians care for higher-risk patients than others).18 Even randomization cannot compensate for a confounded design.20

이같은 연구의 다른 측면과 결과 사이의 긴장은 다음 질문을 떠오르게 한다. '강한 인과적 해석이 가능하지만 결과가 약한 연구와, 결과는 강력하지만 결과 해석에 교란변수가 개입되고 결과가 약한 연구 중 무엇이 나을까?' 답은 상황에 따라 다르겠지만, 많은 경우 인과관계가 더 분명한 것이 더 선호된다. 결과가 개선되었더라도, 그것의 원인을 설명할 수 없다면 기존의 지식에 더할 수 있는 것이 제한된다.

The tensions between outcomes and other aspects of study design raise the following question: Which is better, a study that permits strong causal interpretations (e.g., strong design and limited confounding) with a weak outcome, or a study with a strong outcome but a design that allows for only weak and confounded interpretations? The answer depends on the situation, but in many cases the stronger causal design may be preferable. Being able to state that outcomes improved, without establishing a clear link to what actually caused that improvement, does little to advance the community’s understanding of how to enhance future clinical/educational practice.35

결과 선택의 편항

Potentially biased outcome selection

다음의 일화를 보자

The following anecdote illustrates a fourth limitation of patient outcomes research in medical education:

한 여성이 길을 가다가 가로등 밑에서 무릎을 꿇고 있는 남자를 보았다. 무엇을 하느냐고 물으니 열쇠를 찾고 있다고 했다. 그녀도 같이 열쇠를 찾고자 했으나 찾지 못했고 남자에게 물었다. "여기서 잃어버린게 맞나요?" 남자가 대답했다. "아, 아니요. 저쪽에서 떨어뜨렸는데 거기는 어둡더라구요. 여기가 훨씬 밝아서요."

A woman walking down the street one night noticed a man on his knees under a lamppost. When asked what he was doing, the man replied that he was looking for his keys. She joined him in the search, but after several minutes asked, “Are you sure you lost them here?” “Oh no,” the man replied. “I dropped them on the other side of the street. But it’s dark over there; the light is much better here.”

예를 들면 최근의 systematic review를 보면 procedural task에 대한 시뮬레이션기반교육이  환자결과에 미치는 영향을 보고 있지만, nonprocedural task에 대한 시뮬레이션기반교육이 미치는 영향을 본 연구는 없다.

For example, researchers conducting a recent systematic review of simulation-based education found that all of the studies reporting patient outcomes focused on procedural tasks (e.g., endoscopy and endotracheal intubation),32 whereas no studies used patient outcomes to evaluate simulation-based training activities for no-less-important nonprocedural tasks (e.g., physical exam or crisis resource management).8

관심영역 전체를 잘 반영해주지 못하는 결과를 선택하는 실수를 하기도 한다. 교육분야의 많은 성과 연구들이 그 성과를 직접적으로 측정하는 방법이 없기 때문에 'broad curricular goals'를 가장 잘 반영하는 것보다는 '측정이 용이한' 도구를 선택하곤 한다.

Researchers risk bias when they select an outcome that does not reflect the entire domain of interest. Regrettably, many of the relevant outcomes in education research do not readily lend themselves to measurement,18 and researchers thus select measures that are easy rather than those that best reflect broad curricular goals.

의학교육연구자들은 환자결과를 모니터하는 데 필요한 연구비를 충분히 가지고 있지 못해서, 의무기록을 활용하기도 한다. 의무기록은 수집하기는 쉽지만 실제 수행능력을 잘 보여주는 데이터는 아닐 수 있다. 가능하거나 용이하다는 이유만으로 어떤 성과를 선택하는 것은 비뚤림을 유발할 수 있다.

Medical education researchers often lack funding to prospectively monitor patient outcomes; therefore, they select measures available from the medical record. Although easy to collect, such data may not be good indicators of actual performance.18 Selecting a given outcome for reasons of availability or feasibility introduces possible bias in the clinical topic or the measurement approach.

가로등 및에서 찾는 것이 항상 문제인 것은 아니다. 그러나 임상 결과 뿐만 아니라 일반적으로 평가에 대해서 한 영역에서 우수한 것이 꼭 다른 영역의 우수함과 연관되는 것은 아니다. 예컨대 혈당관리를 잘 하는 것이 대장암 스크리닝도 잘 한다는 의미는 아니다.

Looking under the lamppost is not always a problem. Researchers wishing to demonstrate “proof of concept” might reasonably select a test-case clinical topic with a patient outcome that is intentionally easy to measure. However, as authors have noted for both clinical outcomes18 and assessment in general,36 performance in one domain often has little correlation with another. For example, superior performance in managing blood glucose in diabetes may not predict performance in colon cancer screening.

시험을 잘 보기 위한 교육

Teaching to the test

평가가 학습을 유도한다는 오래된 지혜처럼 많은 교육자들은 여기에 동의할 것이다. 평가는 학습자에게 동기부여를 하거나 교수와 학생 모두 지나치기 쉬운 learning gap을 발견하게 해준다. 그러나 환자중심 성과에 지나치게 몰두하는 것은 역효과가 날 수도 있다.

Conventional wisdom indicates that assessment drives learning, and most educators (including ourselves) would agree that this is usually a good thing. Assessment can motivate and focus both learners and teachers to address learning gaps they might otherwise overlook. However, excessive focus on patient-oriented outcomes could negatively affect teaching by leading educators to “teach to the test.”

환자결과에 대한 과도한 집중은 교육과정개발자들이 '확실히 환자 진료에 도움이 되는 것'만 가르치도록 할 가능성이 있다. 일견 합당해보이지만 두 가지 중요한 문제가 있다.

Too much attention to patient outcomes could lead curriculum designers to teach only those processes that unambiguously enhance patient care. Although seemingly sensible, this approach suffers from at least two shortcomings. 

연구자들이 '상황(situation)'을 개선하려고 노력하지만, '명백한 근거'는 의사가 진단하고 치료결정을 내리는 아주 일부분에 대해서만 알려줄 뿐이다. 미리 정해진 기준에만 맞춰서 집중하는 것은 다른 주제들을 간과할 수 있다.

First, despite the research community’s valiant attempts to improve the situation, clear evidence informs only a fraction of clinicians’ diagnostic and therapeutic decisions.37 Focusing primarily on practices with  defined standards will necessarily detract from teaching on other topics. 

두 번째로, 근거기반 알고리즘 접근법에 집중하는 것은 학습자들로하여금 그러한 행동의 기저에 깔린 원칙을 배우지 않고 넘어가게 만들 수 있다. '시스템'이 교육보다 환자결과에 더 강한 효과가 있을지는 몰라도, 병태생리를 배우고 다른 근본 원칙을 배우는 것은 학습의 retention과 transfer에 중요하다. 또한 이러한 지식을 활용하여 새로운 치료법을 이해하고, 새로운 연구 결과를 해석하고 후에 연구를 주도해나가는 것에도 이득이 된다.

Second, focusing on evidence-based algorithmic approaches to management could backfire if learners fail to learn the principles that underlie such actions. Although systems change usually has a stronger effect on patient outcomes than education,38 learning pathophysiology and other underlying principles has clear benefit on retention and transfer,39 to say nothing of the long-term benefits of such knowledge40 in understanding new therapies, interpreting new study results, or conducting research later in life.

환자 결과 : 항상 좋은 것만은 아니다.

Patient Outcomes: Not Always Better

환자결과가 다른 '결과'보다 우월하다는 주장 역시 개인의 가치에 따른 주관적인 판단일 뿐이다(value judgement). Shea는 교육의 일차 고객은 '학습자'이지 '환자'가 아니다 라고 했다.

The argument that patient outcomes are superior to other outcomes is, ultimately, a value judgment. Shea19 pointed out that “the primary customer of medical education is emphatically the learner, not the patient.” 

지식, 술기, 태도를 측정하는 것, 심지어 '만족도'를 측정하는 것들에 대해 '부차적'이라고 격하할 필요는 없다. 모든 것들이 동등하다면, 만족도가 높은 것이 나쁜 것이 아니다! Yardley와 Dornan이 지적한 것처럼 Kirkpatrick의 낮은 레벨에서 또는 nonoutcome evidence를 통해서 얻을 수 있는 것도 많다. 과도하게 환자결과에 초점을 맞추는 것은 학생들을 'dehumanizing'시킬 수 있다.

However, measures such as knowledge, skills, attitudes, time, and even satisfaction should not automatically be relegated to second-tier status as “process measures.”2 All else being equal, higher learner satisfaction is not a bad objective! Also, as Yardley and Dornan9 have noted, the community can learn much about an educational activity from outcomes lower in Kirkpatrick’s model and from nonoutcomes evidence (e.g., process evaluation and qualitative data). Furthermore, excessive concentration on patient outcomes risks dehumanizing trainees (and thereby the education process) by viewing the trainee solely as a means to an end rather than a worthy end in and of him- or herself.

비-환자결과는 '이론을 수립하는 연구'에서 특히 중요한데, 이러한 종류의 연구가 종종 환자 접촉이 제한적인 상황에서 이뤄지기 때문이다. 의학교육연구는 그 이론이 없다는 것 때문에 항상 아쉬움이 있었는데, 일부 연구자들은 '이론을 수립하는 연구'를 중요하게 본다.

Nonpatient outcomes (knowledge and skills) may be particularly important in theory-building research because this type of research often occurs in settings with limited patient contact. Medical education researchers have frequently lamented the absence of theory in the field,41,42 and some have suggested theorybuilding research as central to advancing the community’s understanding of how to improve learning activities.43

제언 

Recommendations

측정도구를 정하고 그것을 중심으로 연구를 설계하고 수행하기보다는 처음에 연구의 목적과 개념틀을 분명히 하고 가장 관련있는 결과, 측정법, 측정도구를 정해야 한다.

First, rather than starting a research project by identifying a measure or tool (e.g., “hemoglobin A1c” or “the patient record”) and then designing the investigation around it, researchers should first clarify the study objective and conceptual framework, then select the most relevant outcome, then the measurement method, and finally the instrument.44 

By selecting the question first, they both maintain focus on the most important issues and avoid prematurely selecting an outcome or instrument that will not provide the most meaningful data. Researchers must also ensure that the outcomes align with the educational objectives. 

No “most important” outcome exists in absolute terms—only better outcomes for a given context and purpose. The best outcome will balance two (at times opposing) requirements: the need to provide meaningful conclusions for the intended audience and the constraints of feasibility.

환자와 관련된 성과에 대한 토론을 할 때, 좀 더 명확한 토론을 위해서 교육자들은 'skill'과 'behavior'와 'patient effects'를 구분할 필요가 있다.

Second, for purposes of clarity in discussing the patient-related outcomes of health professions education, educators should remember the distinction between skills (provider actions in an artificial test setting), behaviors (provider actions with real patients, such as ordering tests, prescribing, procedural time, or procedural technique), and patient effects (Kirkpatrick’s level 4 “results”: the actual impact on patients, such as patient satisfaction, patient compliance, symptom control, complications, or test results).8,32 

Of note, a patient characteristic such as motivation to change might be considered an attitude in clinical research, but we argue that in health professions education research this characteristic qualifies as a true patient effect.

환자결과와 다른 접근성 높은 결과들의 연결고리를 만들 필요가 있다.

Third, researchers need to focus on establishing links between patient outcomes and other more accessible outcomes. 

To link patient outcomes causally to an educationally relevant activity or personal characteristic can be challenging. If the conceptual relationship between the activity and the outcome is poorly defined, investigators will be unable to bridge the gap with a single link. In such instances, they may find that using two or more links provides a more readily accessible chain of causality. 

For example, if researchers demonstrate an association between specific skills or behaviors and specific patient outcomes, then they and others may use these skills or behaviors as surrogate outcomes in subsequent studies (see Figure 1). 

For example, in a simulation-based course on vascular surgery, investigators found that simulator outcomes of time and severity of anastomotic leaks (skills) were associated with operative time and anastomotic leaks in real patients.45 Another study found an association between the quality of counseling with real patients (a behavior) and the patients’ motivation to change (a patient effect).46 

Of course, surrogate outcomes can be misleading,47 as is well understood in clinical research.48 Adapting existing guidelines for the use of surrogate end points in clinical research to medical education research seems prudent,49 including not only that the surrogate must correlate with the patient outcome but that improvement in the surrogate should also associate with improved patient outcomes

연구자들은 교육적 개입방법을 검증할 때 신중히 여러 단계를 밟아야 한다. 맨 처음 지식과 술기를, 그 다음 행동을, 그 다음 환자 결과를 검증해야 한다.

Fourth, investigators should consider proceeding in a deliberately stepwise fashion as they test educational interventions: first assessing knowledge and skills, then behaviors, and finally patient outcomes. 

우리가 초점을 옮겨가기 전에, 학생의 행동에 영향을 줄 수 있는가를 확실히 해야 한다. 그리고 그 다음으로 가야 한다.

As Shea19 stated, “Before we shift our focus—and simultaneously the expectations of reviewers and editors—we need to make sure we can influence students’ behaviors. Once we know how to do this, we can turn our attention to the next link.” 

의사들에게 있어서 행동과 환자결과 사이의 연결고리는 좀 더 직접적이지만, 아직 수련중인 수련의/전문의에게는 이 연결고리는 덜 직접적이고, 의대생들에게는 더 그렇다.

Researchers should also consider the learner’s training level: The link between behaviors and patient outcomes is much more direct (less diluted) for physicians, and to a lesser extent for postgraduate trainees, than it is for medical students. 

A study of cardiac resuscitation training for internal medicine residents illustrates the stepwise progression of outcomes: First, the investigators established that the course improved resuscitation skills in a simulated setting50; then, in a subsequent study, they assessed behaviors (checklist score during actual resuscitation) and patient outcomes (survival to discharge).51 

Another program of research began with an assessment of the need for training in obesity counseling,52 proceeded with a study evaluating the impact of training on patient counseling activities (behaviors),53 and then evaluated the effect on weight change54 (a patient outcome).

연구자들은 환자결과를 선택할 때 환자 또는 전체 의료팀이 개입된 것을 선택할 수도 있다. Kalet 등은 "educationally sensitive patient outcome"이라는 개념을 통해서 스스로의 치료에 대한 환자의 자발적 참여, 그리고 의료진의 효과적 개입에 영향을 줄 수 있는 개개인의 능력에 관심을 두었다. 

Fifth, investigators might consider selecting patient outcomes that result from the engagement of patients and the whole health care team. Kalet and colleagues16 have offered a conceptual framework for “educationally sensitive patient outcomes” that focuses on the capacity of individual providers to influence patient care by enhancing patients’ active involvement in their own care and by effectively engaging the health care team and available systems. 

These outcomes (e.g., patient motivation to change or team function) lie at the interface between behaviors and patient outcomes. 

In addition, they may offer a feasible approach to studies of educational programs that yield insight into patient care effects— provided educators can develop and implement appropriate measurement tools. The study cited above that linked physician counseling and patient motivation46 illustrates one application of educationally sensitive patient outcomes.

마지막으로, 우리는 훈련자 한 사람당 하나 이상의 환자결과가 나타날 때 advanced statistical techniques 이 필요할 수 있다. clustering을 조절하는 것에 실패하면 study power를 인위적으로 높게 보고 잘못된 결론을 내릴 수 있다.

Finally, we remind researchers that advanced statistical techniques will be required whenever there is more than one patient outcome per trainee (i.e., clustering of patients).30,32 Failure to adjust for clustering when required constitutes a unit-of-analysis error that artificially inflates the study power and may lead to spurious conclusions.

---

Acad Med. 2013 Feb;88(2):162-7. doi: 10.1097/ACM.0b013e31827c3d78.

Perspective: Reconsidering the focus on "outcomes research" in medical education: a cautionary note.

Cook DA1, West CP.

Author information

Abstract

Researchers in medical education have been placing increased emphasis on "outcomes research," or the observable impact of educational interventions on patient care. However, although patient outcomes are obviously important, they should not be the sole focus of attention in medical education research. The purpose of this perspective is both to highlight the limitations of outcomes research in medical education and to offer suggestions to facilitate a proper balance between learner-centered and patient-centered assessments. The authors cite five challenges to research using patient outcomes in medical education, namely (1) dilution (the progressively attenuated impact of education as filtered through other health care providers and systems), (2) inadequate sample size, (3) failure to establish a causal link, (4) potentially biased outcome selection, and (5) teaching to the test. Additionally, nonpatient outcomes continue to hold value, particularly in theory-building research and in the evaluation of program implementation. To educators selecting outcomes and instruments in medical education research, the authors offer suggestions including to clarify the study objective and conceptual framework before selecting outcomes, and to consider the development and use of behavioral and other intermediary outcomes. Deliberately weighing the available options will facilitate informed choices during the design of research that, in turn, informs the art and science of medical education.

Comment in

• In reply. [Acad Med. 2013]
• More about the focus on outcomes research in medical education. [Acad Med. 2013]
• Why medical educators should continue to focus on clinical outcomes. [Acad Med. 2013]
• In reply to Archer and to Lim. [Acad Med. 2013]
• To the editor. [Acad Med. 2013]