CBAS와 어려워하는 전공의 탐색 및 서포트의 관계(JAMA, 2018)

CBAS와 어려워하는 전공의 탐색 및 서포트의 관계(JAMA, 2018)

Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty

Shelley Ross, PhD; Natalia M. Binczyk, BMSc; Deena M. Hamza, PhD; Shirley Schipper, MD; Paul Humphries, MD; Darren Nichols, MD; Michel G. Donoff,MD

도입

Introduction

CBAS는 두 가지 기본에 근거한 프로그램적인 평가로 설계되었다. 하나는 AOL이며, 다른 하나는 레지던트와 공유되는 정기적 형성피드백이다(저부담 평가도구와 함께 문서화된다). CBAS는 직장 기반 교육WBL에서 레지던트에 대한 직접적인 관찰에 초점을 맞추고 있다. CBAS는 학습자를 관찰한 후, WBA의 모범 사례를 준수하여 전문가의 판단과 코칭을 촉진 및 capture하도록 지원합니다. CBAS의 평가 도구는 preceptor들이 레지던트가 업무에서 수행한 것을 묘사하고, 관찰한 것을 가정의학의 상위 역량영역에 맞춰 tag 또는 분류하도록 한다(전문성, 의사소통 기술, 임상적 추론, 의료 지식, 환자 중심 치료, 실행 관리, 절차 기술 및 적절한 프레젠테이션 우선 순위 지정). 평가가 되는 역량은 우리의 사전 CBAS 대 사후 CBAS 코호트에 유사하지만, 명확성과 이해를 높이기 위해 그러한 역량의 설명자가 변경되었습니다.

The CBAS is designed as programmatic assessment26-28 predicated on 2 fundamentals: assessment for learning20,29,30 and regular formative feedback shared with residents (documented with low-stakes assessment tools).17,21,25 The CBAS focuses on direct observation of residents in workplace-based training. In keeping with best practices of workplace-based assessment, CBAS helps to both facilitate and capture experts’ judgment and coaching after observation of learners. The assessment tools in CBAS are designed to allow preceptors to describe what they see the residents do in the workplace and tag or sort their observations according to high-level descriptions of areas of competence in family medicine (professionalism, communication skills, clinical reasoning, medical knowledge, patient-centered care, practice management, procedural skills, and appropriately prioritizing presenting issues).24 Although the competencies being assessed were similar for our pre-CBAS vs post-CBAS cohorts, the descriptors of those competencies were changed to enhance clarity and understanding.

방법

Methods

STROBE를 따름

We adhered to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for reporting observational cohort studies.31

어떤 전공의가 하나 이상의 측면에서 어려움을 겪는지 알아내기 위해 "flag"라 불린 변수를 찾는다.

Three program directors identified variables (referred to as flags) that indicated that a resident was having difficulty with 1 or more aspects of residency training. These variables are defined in the Box.

통계 분석

Statistical Analysis

Results

전공의의 기본 인구통계학적 특성(표 1)은 국제 의료 졸업생 비율을 제외하고 사전 CBAS와 사후 CBAS 코호트가 유사했다.

The basic demographic characteristics of the residents (Table 1) were similar between the pre-CBAS and post-CBAS cohorts with the exception of the proportion of international medical graduates.

종합평가에 있어 깃발을 받은 전공의의  비율의 차이는 표 2에 제시되어 있다.

Differences in the percentage of residents receiving flags on summative assessments are presented in Table 2.

표 3은 주어진 전체 정의에 따라 resident in difficulty 기준을 충족하는 각 코호트 내 전공의 비율 변화를 보고한다.

Table 3 reports changes in the proportions of residents within each cohort who met the criteria for designationas a resident indifficulty, according tothe definitions given.

또한 평가 플래그가 전공의와  함께 해결되었다는 증빙 자료(그림)의 빈도로 CBAS 전 훈련과 사후 CBAS 훈련 간의 변화를 분석하였다.

We also analyzed changes between pre-CBAS training and post-CBAS training in the frequency of evidence of documentation that a flag on an assessment had been addressed with the resident (Figure).

고찰

Discussion

이러한 연구 결과는 CBME로의 전환을 위한 정당성을 보여주며, 특히 CBME가 전통적인 의료 교육보다 개선되었다는 증거의 필요성에 대해 과거 문헌에서 제기된 몇 가지 질문에 답하기 시작한다. CBME로 전환하기 전 프로그램에 사용된 기존의 평가 방식과 비교했을 때, 역량 기반 평가는 훈련에 어려움이 있는 레지던트를 더 잘 식별하였으며, 어떻게 전공의에 대한 concern이 해결되었는지에 대한 향상을 보여주었다.

These findings begin to answer some of the questions raised in the literature about justification for the shift to CBME,8-14 specifically, the need for evidence that CBME is an improvement over traditional medical education. Compared with the traditional assessment approach used in our program before the switch to CBME, competency-based assessment was associated with better identification of residents who encountered difficulties in training and improvement in how concerns about resident competence were addressed.

CBAS를 시행한 이후, 총괄평가에서 최소 1개의 플래그를 받는 레지던트의 비율이 크게 감소했다.

Since implementation of CBAS, there has been a significant decrease in the proportion of residents receiving at least 1 flag on a summative assessment.

여러 개의 flag을 받은 레지던트 비율이 크게 감소했다.

There were large decreases in the proportion of residents who were receiving multiple flags.

복수의 개별 로테이션에서의 flag 감소 및 어려움에 처한 레지던트로 판정되는 레지던트 비율 감소 사이에 잠재적 연관성이 발견되었다. 비록 1개의 로테이션에서 flag를 받은 레지던트 비율은 변화가 없었지만, 2개 이상의 로테이션에서 flag를 받은 레지던트의 비율은 2012년 1명을 제외하고 약 0%로 감소했다.

A potential association was found between the decrease in flags on multiple discrete rotations and the reduction in the proportions of residents who met criteria for resident in difficulty. Although the proportion of residents who received a flagged assessment from1 rotation remained stable across the study period, the proportion of residents who received a flag on assessments from more than 2 rotations decreased to approximately 0%with the exception of 1 resident in the 2012-2014 cohort.

CBAS 방식의 접근법이 전공의에 대한 더 나은 support와 관련되어있을 가능성을 보여두는 또 다른 것은 역량 부족으로 flag된 전공의가 추가적인 support를 받았다는 것이다. 이는 총괄평가에서의 flag로 나타난 경우에 레지던트와의 토론이 더 늘어났다는 것이다. 학습자가 difficulty를 겪는 것으로 확인되면, 이는 학습자와 함께 논의되어야 하나, 이러한 코칭은 facilitated될 필요가 있다. CBAS 이전에는 1개 이상의 flag를 받은 거주자 중에서 35%~40%는 전공의파일에 flag가 해결되었다거나 전공의와 논의되었다는 증거가 없었다(그림).

The likelihood that the CBAS approach to assessment is associated with better support of residents who are flagged for deficiencies in competence is further supported by the finding of an increase in documentation showing that flags on summative assessments were discussed with the resident. Identified difficulties should be discussed with learners, but such coaching needs to be facilitated. Before CBAS, 35%to 40%of the residents who received 1 or more flags had no evidence in their files that the flag had been addressed or discussed with them(Figure).

전반적으로, 본 연구는 CBAS와 같은 역량 기반 평가 프레임워크가, 역량 gap을 가진 레지던트를 더 잘 식별함을 보여준다. 또한 CBAS는 격차가 확인된 경우, 그것을 해소하고 개선하기 위해 레지던트에 대해 더 나은 서포트를 제공하는 것과 관련이 있는 것으로 보인다. 이 레지던트 프로그램의 기존 평가 접근방식은 전공의가 어려움을 겪고 있는 때를 식별하기 위한 프로세스를 마련했지만, 총괄평가가 일상적인 관찰observation과 단절되었기 때문에 그러한 시스템이 효과적이지 않았던 것으로 보인다. 어려움을 겪고 있는 레지던트를 식별하는 데 실패하는 것은 이 전공의 프로그램에만 국한되지 않습니다. 오히려, 이러한 문제는 의학교육의 여러 평가 방식에서 확인되었으며 CBME로의 변화를 정당화하는 핵심 중 하나입니다.

Overall, this study suggests that a competency-based assessment framework such as CBAS is associated with better identification of residents who have competence gaps. Furthermore, CBAS appears to be associated with better support for residents to address and ameliorate identified gaps. Although the previous assessment approach in this residency program had processes in place that were intended to identify when residents were struggling, the system was ineffective, perhaps because summative assessments were disconnected from daily observations. This failure to identify struggling residents is not unique to this one residency program; rather, this problem has been identified across multiple assessment approaches in medical education and is one of the key justifications for moving to CBME.1,15-17

이러한 연구결과는 평가절차의 개선의 결과로 기각될 수 있다. 그러나, CBME 문화에서의 평가는 반드시 달라야 하며, CBAS 접근법은 조사된 거주 프로그램의 평가에 대한 이전의 접근방식과 근본적으로 다르다. 회전의 끝을 포착하는 데 초점을 맞춘 평가와 달리, CBAS 도구, 양식 및 프로세스는 임상 코치와 공유하는 형태적 피드백의 대표 표본을 포함하여 임상 경험 전반에 걸쳐 역량을 향한 진행의 증거를 포착한다. 이러한 저평가들은 학습을 반영하고 육성할 수 있다.

It would be possible to dismiss these findings as being a result of improving processes of assessment. However, assessment in a CBME culture must be different,17,21,37-40 and the CBAS approach is fundamentally different from the previous approach to assessment in the residency program examined. In contrast to assessment that focused on capturing end of rotation judgements, the CBAS tools, forms, and processes capture evidence of progress toward competence across clinical experiences, including a representative sampling of the formative feedback shared by the clinical coaches who work with the resident. These low-stakes assessments may reflect and foster learning.

역량에 대한 총괄평가는 정기적으로 이루어진다. 고부담 ITER은 매 로테이션 종료시 완료됩니다. 고부담의 정기 발달상황 점검은 4개월마다 이루어집니다(이전에는 6개월마다). CBAS 이후의 차이점은 정기 진행상황 검토를 할 때, [현재 역량 향상의 진행]에 대하여, 전공의가 자기성찰을 문서화한 후, 교수 어드바이저(역량 코치)와 전공가 함께 토론하게 된다는 것이다. 이 때, CBAS에서 수집된 저부담평가를 guided self assessment의 기반근거로 사용한다.

Summative assessments of progress toward competence occur regularly. High stakes in training evaluation reports are completed at the end of every rotation. High-stakes periodic progress reviews occur every 4 months (previously every 6 months). The difference after CBAS is that the periodic progress review is now a shared process in which resident self-reflections on progress toward competence are documented and then discussed between the faculty advisor (competence coach) and the resident, with the low-stakes assessments collected in CBAS used as the evidence base for guided self-assessment.41

CBAS 프레임워크의 평가의 투명성과 형성평가의 정기적 제공이라는 특징은 어려움을 겪는 전공의를 조기에 식별할 수 있는 문화를 만들어냈다. 이러한 문화에 기여하는 두 가지 요소는 다음과 같습니다. 

• 역량 향상의 진행에 대한 문서화된 증거의 확산(강점과 약점을 모두 식별할 수 있음) 

• 레지던트의 학습에 대한 정기적인 토론 

레지던트가 최고의 의사가 될 수 있도록 지원하는 문화에서, flag와 같은 격차를 해소하는 것은 낙인을 찍는 것처럼 느껴질 가능성이 낮다. 

The transparent nature of assessment in the CBAS framework, as well as the regular provision of formative feedback, has created a culture in which residents in difficulty can be identified early. Two factors contribute to this culture: the proliferation of documented evidence of progress toward competence (which can identify both strengths and gaps) and the regular discussion of the resident’s learning. Addressing a gap, such as a flag, is less stigmatizing in a culture in which supporting residents to be the best physicians that they can be is the focus of assessment. 

총괄평가에서 전공의를 flagging하는 과정은 바뀌지 않았다. CBAS 도입 후에도, flag는 여전히 레지던트가 역량을 입증하지 못한 1개 이상의 주제가 있음을 의미한다. CBAS전후의 차이점은, 도입 후에는 [역량에 대한 우려]를 종종 로테이션 중에 레지던트와 논의한다는 것이다. 이는, 많은 경우에, 레지던트의 부족한 부분이 로테이션 종료 시에 이뤄지는 총괄평가 전에 해결됨을 의미한다.

The process of flagging a resident on a summative assessment has not changed: before and after CBAS, a flag means that there are 1 or more topics on which a resident has not demonstrated competence. The difference is that concerns about competence are often discussed with the resident throughout a clinical experience, which means that in many cases, deficiencies are remedied before the final summative assessment at the end of the rotation.

Limitations

Conclusions

이 다년간의 역량 기반 평가 및 전통적인 평가 구현의 비교에서 얻은 결과는 CBME의 개념 증명(Proof of Concept)을 지원합니다. 평가의 초점을 

• 직접적인 관찰에 대한 강조 

• 문서화 증가

• 학습을 위한 평가 

에 두는 것은 [1개 이상의 역량이 부족한 학습자를 발견하고], [그 부족을 어떻게 해결할 것인가]와 관련이 있다

The findings from this multiyear comparison of implementation of competency-based assessment and traditional assessment support a proof of concept for CBME. Changing the focus of assessment to an emphasis on direct observation, increased documentation, and assessment for learning may be associated with improved identification of learners who are deficient in 1 or more competency and with how those deficiencies are addressed.

---

Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty

Shelley Ross, PhD¹; Natalia M. Binczyk, BMSc¹; Deena M. Hamza, PhD¹; et alShirley Schipper, MD¹; Paul Humphries, MD¹; Darren Nichols, MD¹; Michel G. Donoff, MD¹

Author Affiliations Article Information

JAMA Netw Open. 2018;1(7):e184581. doi:10.1001/jamanetworkopen.2018.4581

Key Points

Question  Is competency-based assessment associated with changes in rates of identification of and support for residents in difficulty compared with traditional assessment?

Findings  In this cohort study of 458 Canadian medical residents, there were significant reductions in the proportions of residents receiving flagged assessments on multiple rotations, reductions in proportions of residents defined as being in difficulty, and increases in documented evidence identifying that gaps were discussed with the resident following introduction of a competency-based assessment program.

Meaning  Competency-based assessment may contribute to better identification of and support for residents in difficulty.

Abstract

Importance  Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools.

Objective  To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty.

Design, Setting, and Participants  This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada.

Exposures  Introduction of the Competency-Based Achievement System (CBAS).

Main Outcomes and Measures  Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program.

Results  Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183).

Conclusions and Relevance  The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program’s ability to address learners’ deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.