성찰 능력 촉진과 평가: REFLECT 루브릭 개발(Acad Med, 2012)

성찰 능력 촉진과 평가: REFLECT 루브릭 개발(Acad Med, 2012)

Fostering and Evaluating Reflective Capacity in Medical Education: Developing the REFLECT Rubric for Assessing Reflective Writing

Hedy S. Wald, PhD, Jeffrey M. Borkan, MD, PhD, Julie Scott Taylor, MD, MSc,

David Anthony, MD, MSc, and Shmuel P. Reis, MD, MHPE

의학 교육 내에서 성찰 능력을 키우는 것은 다음에 도움이 된다.

• 비판적 사고 능력 개발,

• 임상 추론,

• 전문직업성 향상

성찰은 핵심 프로페셔널 실무 역량에 필수적입니다.

성찰은 개인의 경험을 실천에 inform하는 전문성-강화적, 메타인지적, 암묵적 프로세스이다.

Fostering reflective capacity within medical education helps develop critical thinking skills,1,2 inform clinical reasoning,3 and enhance professionalism4 among trainees. Reflection—the expertise-enhancing, metacognitive, tacit process5,6 whereby personal experience informs practice7—is integral to core professional practice competencies.8,9

성찰 능력의 개발은 의학 교육에서 피드백을 효과적으로 사용하기 위해 필요에 따라 강조되어 왔으며, 자기 조절과 평생 학습의 필수적인 측면이다 .5,10 성찰은 임상 실습에 내재 된 복잡성에 직면 할 때 지침을 제공 할 수있다. 성찰은 "어렵거나 도덕적으로 모호한 상황"에서 행동하는 방법의 선택에 잠재적으로 영향을 미친다 .12 이러한 맥락에서 성찰적 실천을 발전시키는 것은 개인의 성격이나 "미덕"을 고양시켜 "마음의 습관habit of mind", "성향적 경향dispositional tendency", "의학적 도덕성medical morality"을 키우는 것과 관련되어있다. 이것들은 임상 추론, 윤리, 가치관과 관련하여 발생할 수 있는 딜레마에 접근하는데 영향을 준다. 성찰은 또한 "phronesis"를 개발하는데 도움이 된다. 실천적 지혜란 적응적 전문성adaptive wisdom 또는 실천적 지혜practical wisdom로서 전문적이고 역량있는 의료행위를 guide하는 것이다 .13,17 자신의 사고 프로세스를 성찰하지 못하는 것(가정, 신념, 결론에 대한 비판적인 검토를 포함)은 최근 의학에서의 진단 오류의 원인이되는 "의사의 과신physician overconfidence"의 인지 구성 요소이다.18

Development of reflective capacity has been highlighted as necessary for effective use of feedback in medical education10,11 and is an essential aspect of self-regulated and lifelong learning.5,10 Reflection can guide practitioners as they encounter the complexity that is inherent to clinical practice, potentially influencing the choice of how to act in “difficult or morally ambiguous circumstances.”12 In this vein, the development of reflective practice has been associated with enhancing an individual’s character or “virtue,” fostering a “habit of mind,”13 “dispositional tendency,”14 or “medical morality”15 with which to approach clinical reasoning and ethical or values- related16 dilemmas that may arise. It also helps in developing “phronesis”— adaptive expertise or practical wisdom to guide professionally competent clinical practice.13,17 Failure to reflect on one’s own thinking processes, including critical examination of one’s assumptions, beliefs, and conclusions, was recently described as a cognitive component of “physician overconfidence,” a contributing cause of diagnostic error in medicine.18

성찰 능력에 대한 정의는 무척 많은데, 일반적으로 미래의 행동을 이끌기 위한 검토, 해석, 이해 경험을 포함한다.

Definitions of reflective capacity abound, though they generally include review, interpretation, and understanding experiences to guide future behavior.

• 만 (Mann)과 동료 연구원은 성찰 능력을 "더 깊은 의미와 이해를 이루기위한 지식과 경험의 비판적 분석"이라고 정의한다.

• 성찰 능력의 이론적 기둥에는 다음이 포함된다.

• Schon's의 진보: 

• Knowing in action

• 놀람 

• 행동 중 성찰(thinking on our feet), 

• 실험, 

• 행동 후 성찰 (postexperience reflection)

• Boud 등: 성찰에서 감정을 다루는 것에 중점을 둡니다.

• Moon 23은 학습에서의 성찰을 위한 meaning making의 구성요소를 소개한다.

• Mezirow24는 premise reflection을 transformative or confirmative learning과 연결하여 성찰의 개념화에 깊이와 폭을 더합니다.

• Mann과 동료는 성찰 모델에서 반복적이고 수직적 인 두 가지 차원을 기술한다. 

• 반복적 차원은 경험에 의해 유발 된 것으로서, 새로운 이해를 일으킨다. 

• 수직적 차원은 표면적 (기술적) 수준과 더 깊은 (분석적) 수준의 성찰을 결합합니다.

• Mann and colleagues19 define reflective capacity as “critical analysis of knowledge and experience to achieve deeper meaning and understanding.” 

• Theoretical pillars of reflective capacity include 

• Schon’s20 progression from knowing-in-action, to surprise, reflection-in-action (“thinking on our feet”21), experimentation, and, finally, reflection-on-action (postexperience reflection), and 

• Boud and colleagues’22 emphasis on addressing feelings in the reflective process. 

• Moon23 introduces the component of meaning making to reflection in learning, and 

• Mezirow24 links premise reflection with transformative or confirmatory learning, bringing additional depth and breadth to reflection conceptualization. 

• Mann and colleagues19 describe two overarching dimensions in models of reflection: iterative and vertical. 

• The iterative dimension of reflection is one triggered by experience, producing a new understanding; 

• the vertical dimension combines surface (descriptive) and deeper (analytic) levels of reflection.

성찰이 반드시 직관적인 것은 아니며, 특히 학생들의 의료 진로 초기에 그러하다. 따라서 의학 교육자들은 훈련 과정 초기에 성찰 능력 개발을 촉진하기 위해 혁신적인 교육 방법을 구현하려고 노력합니다. 성찰 연습을 용이하게하기 위해 반사 쓰기 (RW)를 사용하는 것이 잘 문서화되어 있습니다.

Reflection is not necessarily intuitive, especially in students at initial stages of their medical careers. Thus, medical educators strive to implement innovative educational methods to promote development of reflective capacity early in the training process. The use of reflective writing (RW) to facilitate reflective practice is well documented.25–29

"상호작용적" RW- 학습자가보다 정교한 성찰 기술을 개발할 수 있도록 교사로부터 서면 피드백을 통합하는 것 .29

“interactive” RW— integration of written feedback from faculty to foster learners’ development of more sophisticated reflection skills.29

내러티브 의학의 하위 집합 인 RW는 자기 인식을 배양하고 RW와 임상 실습간에 공유되는 참석, 대표, 제휴 과정을 통해 clinical encounter를 위한 서사 역량을 구축한다. RW는 실용적 의학적 추론의 "해석적이고 내러티브적인"31 자질을 구현한다. RW를 통해 배양될 수있는 서사적 역량과 정서적 자기 성찰 능력은 정서적으로 어려운 상황에 대한 회복탄력성을 강화하고 나쁜 소식을 전파하는 등 의사 소통에 어려움을 겪을 수있는 능력을 증진시킬 수 있습니다.

RW, a subset of narrative medicine, cultivates self- awareness and builds narrative competence for clinical encounters through the processes of attending, representing, and affiliating that are shared between RW and clinical practice.25 RW embodies the “interpretative and narrative”31 qualities of practical medical reasoning. Narrative competence and emotional self-reflective ability, which may be cultivated through RW, can bolster resilience to emotionally challenging situations32 and promote capability in challenging communication encounters, such as breaking bad news.33

학습자를 능숙하게 지원하고 challenge하여 성찰 능력을 개발시켜주기 위해서 멘토는 다음을 필수적으로 갖추어야 한다.

• 성찰적 순간을 알아 차리고,

• 경험에 대한 이해 (정서적 반응 포함),

• 불확실성을 용인 ("전문적 지식의 핵심"이 되는 임상 실습의 "지저분함")

• 새로운 통찰력을 사용

Mentors who skillfully support and challenge learners through 

• noticing the reflective moment, 

• making sense of the experience (including emotional responses), 

• tolerating uncertainty (or “messiness” of clinical practice at the “heart of professional expertise”20), and 

• using new insights5,34 are an essential component to developing reflective capacity. 

성찰적 내러티브에 대한 guided written 피드백은 보다 심층적인 성찰 과정을 촉진 할 수있다 .35,36

Their guided written feedback about reflective narratives can promote a more in-depth reflective process.35,36

성찰의 정도와 수준을 결정하기 위해 written reflective 저널을 분석하기 위해 사용 가능한 코딩 시스템을 적용 할 때 중요한 한계와 문제점이 있습니다.

There are significant limitations and challenges in applying available coding systems for analyzing written reflective journals to determine the extent and level of reflection.

의대생의 전문성 개발 및 최상의 교육 관행의 지표로서 반영 수준에 대한 공식적인 평가에 대한 관심 증가에 비추어,우리는 기존의 정성적 및 양적 척도와 성찰 능력을 위한 프레임 워크에 대한 검토를 통해 경험적으로 테스트되고 간결한 "사용자 친화적 인"평가 패러다임을 설계하려고했습니다.

In light of the increased interest in formal assessment of level of reflection as an indicator of professional development of medical students and best teaching practices,35 we set out to design an empirically tested, concise, “user- friendly” evaluative paradigm stemming from our review of existing qualitative and quantitative measures and frameworks for reflective capacity.

방법과 결과

Method and Results

예비 단계

Preliminary stage: Literature and model review

스노우볼 기법 활용

We then used snowball technique to extend the literature search from retrieved articles to other relevant sources. The snowball technique for sampling is a method whereby existing study participants suggest, recruit, or assist in recruiting future subjects from among their acquaintances or contacts.55

문헌 검토에서 네 가지 modality를 도출

From our review of the literature, we identified four existing modalities of reflection assessment: 

• (1) scales (“paper and pencil” forms with responses scored by respondents), 

• (2) thematic coding (qualitative analysis that codes themes in the narratives), 

• (3) qualitative analysis (more elaborate qualitative analysis moving beyond themes into models), and 

• (4) analytical instructional rubrics (theory-based delineation of dimensions or levels of an assessed construct).

위의 네 가지 접근법을 검토함.

We next examined these four approaches for their utility in the assessment of medical students’ RW. Our deliberations were based on both theoretical and functional premises.

• Scale: Although our literature search uncovered an existing scale for measuring “personal reflection,”54 we did not use it for our analysis given its intended purpose for students’ self- reported reflective capacity rather than for assessment of the construct within RW. 

• Thematic coding: Thematic coding26,27 with sole emphasis on extraction of themes was also inadequate for our evaluative aims because students’ reflective levels within RW could not be determined with such a method. 

• Qualitative analysis: Similarly, qualitative analysis was deemed insufficient because of its inability to provide focused differentiation of reflective levels. 

• Analytical instructional rubric:
  네 번째 접근법인 분석적 지도 루브릭은 특별히 평가에 사용됩니다. Analytical instructional rubrics 은 이론적 틀을 기반으로하고 특정 목적을 위해 맞춤식으로 작성 될 수 있기 때문에 반영 수준 평가에 가장 적합한 선택 인 것으로 보입니다. Analytical instructional rubrics 은 평가 된 구조의 다양한 차원 또는 수준을 정의하고 각각에 대한 벤치 마크를 정의하며 양적 점수를 산출 할 수 있습니다. (형성평가와 총괄평가에 사용되는) 루브릭 형식은 다를 수 있지만 공통된 기능으로는 strong to weak work product에 대한 연속성에 대한 품질 수준 그라데이션뿐 아니라 프로젝트를 완료하거나 기술 습득시 비교적 복잡한 목록 또는 "무엇이 중요"한지도 포함됩니다.

  The fourth approach, the analytical instructional rubric,56 is specifically used for assessment. Analytical instructional rubrics seemed to be the best choice for the assessment of reflective levels because they are based on a theoretical framework and can be tailor-made for specific purposes. An instructional rubric delineates the various dimensions or levels of an assessed construct, defining benchmarks for each, and can yield quantitative scores.57,58 The rubric format—used for both formative and summative purposes—may vary, though common features include quality level gradations on a continuum of strong to weak work product, as well as a relatively complex list of criteria or “what counts” in completing a project or mastering a skill.59

Iterative development of the initial rubric

Once we had determined which approach to use, we began the process of developing an actual analytical instructional rubric to assess students’ reflective narratives.

The first iterative cycle: Initial reflection rubric.

The second iterative cycle: The REFLECT rubric.

Third iteration.

Present iteration.

'레벨'을 보여주는 숫자를 생략하여 총괄평가보다는 형성평가 목적으로 사용을 유도하고자 하였음.

Given our primary emphasis on analyzing quality of reflection within RW in a developmental context, we decided to omit assigned numbers for reflection “levels” to encourage use of the rubric for formative rather than summative purposes (Appendix 1).

REFLECT rubric application

REFELCT 루브릭을 적용하는 네 단계

The process of applying the REFLECT rubric to a reflective narrative consists of four steps:

1. 전체 서사를 읽으십시오.

2. 단편화 (Fragmentation) : 서사의 세부 사항(구 / 문장)으로 Zoom in하여 모든 기준의 존재와 퀄리티를 평가합니다 (부록 1 참조). 각 기준이 나타내는 수준을 결정하십시오.

3. 게슈탈트 (Gestalt) : 서술의 전반적인 게슈탈트로 zoom out한다(단계 2의 세부 분석을 고려함). 내러티브 전체가 달성하는 수준을 결정합니다. Critical reflection 수준이 달성되면 학습 결과 중 하나 또는 두 가지 모두 (변형 또는 확증 학습)도 달성되었는지 여부를 결정합니다.

4. 본문의 사례를 통해 수준 및 학습 성과 할당을 defend하십시오. "행간을 읽지 마십시오."

1. Read the entire narrative.

2. Fragmentation: Zoom in to details (phrases/sentences) of the narrative to assess the presence and quality of all criteria (see Appendix 1). Determine which level each criterion represents.

3. Gestalt: Zoom out to consider overall gestalt of the narrative (while taking into consideration the detailed analysis of Step 2). Determine which level the narrative as a whole achieves. If the Critical Reflection level is achieved, determine whether either or both learning outcomes (transformative or confirmatory learning) were also achieved.

4. Defend the assignment of level and learning outcomes with examples fromthe text. Do not “read between the lines.”

A sample reflective narrative and REFLECT rubric analysis is presented in Appendix 2. 

Another example can be seen in Supplemental Digital Appendix 1, http://links.lww.com/ACADMED/A68.

통계 분석

Statistical analyses

We applied single-measure ICCs 61 to all datasets and computed ICCs for each iteration of the REFLECT in the pilot developmental phases (Table 1).

ICC를 사용

An ICC is used to measure ordinal/continuous data for interrater reliability for two or more raters when data may be considered interval. It may also be used to assess test–retest reliability. An ICC may be conceptualized as the ratio of between-groups variance to total variance. In single-measure reliability, individual ratings constitute the unit of analysis (i.e., single-measure reliability provides the reliability for a single judge’s rating). Single-measure ICC is the more conservative estimate and can represent how much agreement one rater will have with other raters. We chose to use ICCs because the levels in rubric iterations are ordinal data where gradations are interpretable, with no “natural zero.” Each application of the developing rubric involved at least three raters.

고찰

Discussion

RW 교육은 의료 교육자가 학생들이 성찰적 임상가가 되도록 요청할 때 번성했다 .3,62

RW initiatives within medical education have prospered as medical educators are called on to prepare students to become reflective clinicians.3,62

서면 에세이 방법론은 공감, 개인 성찰, 전문성과 같은 중요한 역량을 활용할 수 있지만 RW의 효과적인 평가는 어려울 수 있습니다.

Although written essay methodology may tap into important competencies such as empathy, personal reflection, and professionalism, effective assessment of RW can be challenging.32

REFLECT 루브릭은 현재 AMS 내에서 닥터 과정과 가정의학 실습 내의 구조화 된 RW 패러다임에 사용됩니다.

The REFLECT rubric is currently used within AMS for structured RW paradigms within the Doctoring course and family medicine clerkship,

교수진은 연구 목적으로 전반적인 "수준"을 평가하지만, 학생들은 이 정보를 피드백으로받지 못합니다. 최근의 근거는 글쓰기의 질과 성찰적 내용간에 유의한 관계가 없음을 보여주므로, 교수진은 글쓰기의 질을 평가하지 않습니다.

Faculty assess overall “level” of reflection for research purposes, but students do not receive this information as feedback. Faculty do not assess quality of writing, in keeping with recent evidence of a lack of significant relationship between quality of writing and reflective content.63

루브릭 개발 프로세스는 다음을 포함한다

• 문헌에 더 깊이 관여하면서 파일럿 루 브릭을 정제하고,

• 루브릭을 다양한 데이터 세트에 적용

• 특정 기준에 합의가 이루어질 때까지 토론.

현재의 반복에서의 ICC 점수는 허용 가능한 interrater 신뢰도를 보여줍니다. 교수개발 워크샵 및 학생지도에서의 피드백을 기반으로 할 때, 평가자와 학생에게 scoring의 feasibility와 acceptability는 유망합니다.

The process of rubric development involved 

• refining a pilot rubric through further immersion in the literature, 

• application of the rubric to various datasets, and 

• discussion until consensus was reached on specific criteria. 

The ICC scores at the present iteration demonstrate acceptable interrater reliability. Feasibility of scoring and acceptability to both raters and students are promising based on feedback from faculty development workshops and use in student instruction.

표준화에 대한 우리의 노력이 유망한 정신측정학적 특성을 가져 왔지만, 총괄평가보다는 형성평가로 루브릭을 사용하는 것이 좋습니다. 성찰에 대한 공식적인 접근 방식을 장려함으로써 성찰의 authenticity가 결여 될 위험이있는 "정량화"또는 "채점"과는 대조적으로, 우리는 REFLECT 루 브릭을 교육자가 성찰 능력의 개발을 평가하고 학생들의 성찰적 내러티브에 건설적이고 개별화 된 피드백을 산출하는 데 도움을 줄 수 있는 정성적 앵커를 제공한다고 생각합니다.

Although our efforts at standardization have yielded promising psychometric properties, we recommend using the REFLECT rubric for formative rather than summative assessment. In contrast to “quantifying” or “grading,” which may risk a lack of reflective authenticity by encouraging more formulaic approaches to reflection,67 we envision the REFLECT rubric as providing qualitative anchors to help educators both assess development of reflective capacity dimensions and formulate constructive, individualized feedback to students’ reflective narratives. 

현재 우리는 학습자의 양적 및 총괄 평가에 사용되는 루브릭에 대한 요구에 반대합니다 .68 이러한 사용은 잠재적으로 상호작용적 RW 내에서 성찰 능력 개발을 저해하고 역효과를 유발할 수 있으므로주의해야합니다.

At this time, we counter calls for rubrics to be used for quantitative and summative assessment of learners.68 We urge caution in this regard because such use may prove counterproductive, potentially inhibiting the development of reflective capacity within interactive RW.

Conclusions

RW 및 그 평가는 의사의 전문성 개발에 대한 이해를 높이고 이 과정을 지원하기위한 교육학 이니셔티브를 지원할 수 있습니다. 효과적인 환자 치료 (자기 인식, 공감, 통찰력) 및 의사의 복지에 대한 전문성의 차원뿐만 아니라 반사를 포함한 메타인지 능력은 잠재적으로 RW 운동을 통해 촉진 될 수 있습니다 .30

RW and its assessment may enhance our understanding of the professional development of physicians and help guide pedagogic initiatives aimed at supporting this process. Metacognitive skills including reflection as well as dimensions of professionalism in effective patient care (such as self- awareness, empathy, and insight), and physician well-being can potentially be fostered through RW exercises.30

50 O’Sullivan PS, Aronson L, Chittenden E, Niehaus B. Reflective ability rubric and user guide. MedEdPortal. August 26, 2010.

53 Devlin MJ, Mutnick A, Balmer D, Richards BF. Clerkship-based reflective writing: A rubric for feedback. Med Educ. 2010;44:1143–1144.

43 Reis SP, Wald HS, Monroe AD, Borkan JM. Begin the BEGAN(The Brown Educational Guide to the Analysis of Narrative)—A framework for enhancing educational impact of faculty feedback to students’ reflective writing. Patient Educ Couns. 2010;80:253– 259.

---

Acad Med. 2012 Jan;87(1):41-50. doi: 10.1097/ACM.0b013e31823b55fa.

Fostering and evaluating reflective capacity in medical education: developing the REFLECT rubric for assessing reflective writing.

Wald HS1, Borkan JM, Taylor JS, Anthony D, Reis SP.

Author information

1

Warren Alpert Medical School of Brown University, Providence, Rhode Island 02860, USA. hedy_wald@brown.edu

Erratum in

• Acad Med. 2012 Mar;87(3):355.

Abstract

PURPOSE:

Reflective writing (RW) curriculum initiatives to promote reflective capacity are proliferating within medical education. The authors developed a new evaluative tool that can be effectively applied to assess students' reflective levels and assist with the process of providing individualized written feedback to guide reflective capacity promotion.

METHOD:

Following a comprehensive search and analysis of the literature, the authors developed an analytic rubric through repeated iterative cycles of development, including empiric testing and determination of interrater reliability, reevaluation and refinement, and redesign. Rubric iterations were applied in successive development phases to Warren Alpert Medical School of Brown University students' 2009 and 2010 RW narratives with determination of intraclass correlations (ICCs).

RESULTS:

The final rubric, the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT), consisted of four reflective capacity levels ranging from habitual action to critical reflection, with focused criteria for each level. The rubric also evaluated RW for transformative reflection and learning and confirmatory learning. ICC ranged from 0.376 to 0.748 for datasets and rater combinations and was 0.632 for the final REFLECT iteration analysis.

CONCLUSIONS:

The REFLECT is a rigorously developed, theory-informed analytic rubric, demonstrating adequate interrater reliability, face validity, feasibility, and acceptability. The REFLECT rubric is a reflectiveanalysis innovation supporting development of a reflective clinician via formative assessment and enhanced crafting of faculty feedback to reflective narratives.

Comment in

• Commentary: a sense of story, or why teach reflective writing? [Acad Med. 2012]
• Commentary: "I hope i'll continue to grow": rubrics and reflective writing in medical education. [Acad Med. 2012]

PMID:

 

22104060

 

DOI:

 

10.1097/ACM.0b013e31823b55fa