재교육의 어려움: 이론적 방법론적 통찰 (Med Educ, 2013)

재교육의 어려움: 이론적 방법론적 통찰 (Med Educ, 2013)

The remediation challenge: theoretical and methodological insights from a systematic review

Jennifer Cleland,1 Heather Leggett,2 John Sandars,2 Manuel J Costa,3 Rakesh Patel,2,4 & Mandy Moffat,1

도입

INTRODUCTION

의학 학위를 받기 위한 시험에서(의과대학 시험에서)의 성취도가 면허시험에서의 수행능력과 졸업 후 임상역량과 관계됨을 보여준 바 있다. 의과대학때 잘 못했던 학생들은 의사가 되어서도 그러하다.

Research has identified that measures of attainment in medical degree examina- tions can predict subsequent performance on licens- ing examinations and clinical competence after medical school.1,2 In addition, weak medical students go on to become weak doctors,3,4

struggling students는 가이드나 서포트를 계속 잘 받지 못할 수 있고 의사들은 종종 underperformance에 fail 주는 것을 꺼려한다. 따라서 학습에 문제가 있는 학생들은 계속 그상태가 해결되지 못한 상태로 남게 되며, 반복적인 실패와 underperformance를 겪는다. 적절한 시점에 나쁜 performance에 대해서 개입하는 것은 해로운 학습 및 행동 패턴을 가진 학생들을 임상에서 문제를 일으키기 전에 즉각적으로 손볼 수 있게 해준다.

struggling students may con- tinue with little guidance or support7 and supervising clinicians are often reluctant to fail underperfor- mance.8–10 Thus, students’ learning problems remain unaddressed, leading to repeated failure and under- performance.5,6,11 Timely intervention for poor per- formance has the potential to enable the individual to deal with adverse learning and behaviour patterns promptly before these cause problems in clinical practice. 

대부분의 재교육 프로세스는 세 단계를 밟는다. 문제 학생의 진단 혹은 발견 / 재교육 인터벤션 / 재시험. 하지만 이 재교육 프로세스는 교수들에게 상당한 시간을 투자할 것을 요구하며, 더 문제는 교수들은 재교육 인터벤션의 효과에 대해서 확신이 없다는 점이다.

Most remediation processes consist of three steps: identification or diagnosis; a remediation intervention, and retesting.12–14 However, these remediation processes place substantial time de- mands on faculty staff.7,13 Furthermore, faculty members report uncertainty about the efficacy of remediation interventions.13

이러한 결과는 지필고사 혹은 임상시험에서 구체적인 재교육 인터벤션이 있기 전과 후의 수행능력을 평가하여 그러한 인터벤션이 효과가 있었다고 결론내리는 다수의 연구들과 배치되는 결과이다. 그러나 "single studies는 그로부터 발견한 지식을 개념/인구집단/세팅/시점에 대해서 일반화하기에 제한적이며, 광범위한 퍼즐의 일부만 보여줄 뿐이다'

These findings are at odds with those of a number of studies that have evaluated performance on written or clinical examinations before and after a specific remediation intervention7,14–38 and concluded that the intervention was effective. However, single studies ‘are limited in the generalisability of the knowledge they produce about concepts, populations, settings and times’ and ‘frequently illuminate only one part of a larger explanatory puzzle’.39

방법

METHODS

연구 선택

Study eligibility and selection

 

자료 추출

Data extraction

우리는 TREND 체크리스트를 활용함. 그 이유는 이것이 활용한 이론의 보고, 인터벤션의 묘사, 여러 조건의 비교, 연구 설계 등을 강조하기 때문이다.

We used the TREND (transparent reporting of evaluations with non-randomised designs) checklist48 to guide data extraction. This was selected as it emphasises the reporting of theories used and descriptions of intervention and comparison con- ditions, and research design, in evaluation studies that use non-randomised designs, and so was felt to be appropriate for review given the nature of the research studies on the topic.

 

우리는 우리의 주관성을 인정하고, 협동적 탐구cooperative enquiry 원칙을 적용함(관찰결과에 대한 토론, 비판적 성찰과 확장)

We explicitly acknowledged our subjectivity and used the principles of cooperative enquiry (i.e. discussing findings, and critically reflecting and expanding on them50) on an ongoing basis to address this within the group.

 

자료 통합과 분석

Data synthesis and analysis

qualitative synthesis of the data 를 했음

A qualitative synthesis of the data was selected as this method was deemed appropriate in the context of our research questions (Evidence for Policy and Practice Information and Co-ordinating Centre [EPPI-Centre; http://eppi.ioe. ac.uk/cms]49).

결과

RESULTS

대부분의 연구는 연구대상자가 적었다.6-377, 중간값 23. 표본크기를 계산한 연구는 단 하나.

Most studies included very small sample numbers, ranging from six to 377, with a median of 23. Only one study stated a sample size calculation.55

대부분의 연구는 (인터벤션) 이후에 수행된 구체적인 시험에서의 결과에 초점을 두었다(USMLE 등). 실제로 많은 연구들은 공공연하게 (재시와 같은) 구체적인 시험을 위한 시험의 기술과 내용에 초점을 맞추고 있었다.

Most (n = 22, 71%) studies focused solely on performance on a specific, subsequent examination (e.g. resits, re-taking a standard examination such as the US Medical Licensing Examination, or the next standard examination in a programme).14–22,24,26–30,32–35,37,51,52 Indeed, many studies overtly focused on examination technique and content boosting for a specific examination (which the participants were required to re-sit or re-take to progress their education or training).

8개 연구는 재교육에 있어서 보다 holistic한 관점을 유지하였는데, personal support를 제공하였거나, 지식과 스킬의 교수-학습에 대한 보다 포괄적인 접근법을 취했다.

The remaining eight studies (26%) took a more holistic perspective to remediation and either in- cluded the provision of personal support or took a broader approach to learning and teaching skills and knowledge.7,23,25,31,36,38,53,54

복잡성complexity에 대해서도 평가했다. 다수의 상호작용적 요소를 활용한 경우 complex 한 인터벤션으로 정의했다. 대부분의 연구는 구체적으로 무엇을 했는지, 얼마나 길게, 왜 했는지 자세하게 기술하지 않았다. 대부분의 인터벤션은 다양한 활동을 했음을 보고했으나(e.g. tutorials, directed reading, skills practice, feedback, examination prac- tice, case presentations), 그 접근법을 적절한 이론에 따라 정당화한 연구는 거의 없었다. 예를 들면 대부분의 연구는 단순히 프로그램 요소가 멘토링이나 튜터리얼을 포함했다고만 쓰고, 그 내용이나 형태, rationale 등을 쓰지 않았다.

Studies were assessed for complexity.46 An interven- tion was defined as complex if it utilised several interacting components. Dimensions of complexity can include, for example, the number of and inter- actions between components, number and variability of outcomes and the degree of flexibility or tailoring of the intervention permitted. The majority of the studies reviewed did not report in detail what they did, why and for how long. Most interventions reported a variety of activities (e.g. tutorials, directed reading, skills practice, feedback, examination prac- tice, case presentations), but few studies clearly justified their approaches on the basis of appropriate theory (with the notable exceptions of 14,18,19,30,53). Many studies simply stated, for example, that the programme elements included mentoring or tutorials without describing the content or format of, or rationale for, these tutorials or mentoring ses- sions.17,34,35

 

 

고찰

DISCUSSION

대부분의 연구는 근거의 퀄리티를 보면 퀄리티가 낮았다. 이는 주로 통제되지 않은 사전-사후 연구로, 소수의 학생을 대상으로 한 것, 장기 성과 척도가 없는 것 등이 원인으로, 인터벤션의 효과를 호손효과 혹은 background effect와 구분하기 어렵게 한다.

This review established that the addressing of underperformance in medical students or doctors in training is an active area of primary research, but the majority of studies identified would be classed as being of low quality according to the criteria for grading quality of evidence.39 The evidence comes predominantly from uncontrolled before-and-after studies with small samples and few process or long- term outcome measures, which may not convincingly distinguish intervention effectiveness from back- ground effects or the Hawthorne effect.56,57

복잡성에 대한 이슈는 명확하다. 연구의 설계나 방법을 보면 디테일이 부족했다. 구체적으로 어떤 요소가 차이를 만들어냈는지를 밝히는데 도움이 되지 않았다.

The issue of complexity is clear. The designs and methods of the studies reviewed, and the lack of detail reported on the precise nature of many of the interventions, do not allow us to identify which components of the process actually made a differ- ence.

 

일반적으로 인터벤션은 '같은 것을 한번 더 하는 것'인 경향이 많았다. 원래의 교수법이 배우는데 적절한 도움이 되지 않았으면, (비록 시니어 교수에 의해 소그룹으로 진행되었다 해도) 같은 것을 한 번 더 하는게 두 번째에 도움이 될 것이라고 볼 이유가 희박하다.

Generally, interventions tended to represent ‘more of the same’, such as additional or intensive knowledge or skills teaching. If the original teaching did not help students to learn appropriately, there seems little reason to assume that ‘more of the same’ will do so a second time around, even if this is delivered in small groups by senior faculty staff.

 

학생들을 서포트해서 다음 단계로 넘어가게 만드는 것의 윤리성은 그들의 poor한 수행능력을 지속하게 만든다는 점에서 좋게 보아도 의문점이 많을 뿐이다. 또한 제한된 교수의 자원이 향상이 없는 progression을 지원하는데 사용되어야 하는지에 대해서도 논란이 있으며, 교수들이 senior 학생일수록 fail시키기 어려워한다는 여러 근거를 종합할 때, weak student가 weak doctor가 될 가능성이 높기 때문이다.

The ethics of sup- porting students to progress to the next stage of training only to continue to perform poorly (e.g. 6,15,38 ) are, at best, questionable. It is also debatable whether scarce faculty resources should be used to support progression without improvement, which may take weak students further towards registration as potentially weak doctors,3,4 when the evidence sug- gests that faculty members find it harder to fail senior students.8

 

더 나아가, 우리는 어떤 종류의 추가적 서포트가 효과가 있었으며, 얼마나 추가적인 교육이 중요한지 모른다. '만족도'에 대한 측정은 우리의 이해에 별로 기여하는 바가 없다. 그러나 어떤 연구들은 무엇이 차이를 만들어냈는가를 밝히고자 했으며, 여기서 특정 서브그룹이 재교육에 가장 잘 반응한다는 것을 암시한다.  

Moreover, we do not know what types of extra support work, or how much extra teaching is critical. The process measure of ‘satisfaction’, where em- ployed, does not add much to our understanding of barriers, facilitators or what precisely works. However, those studies that explicitly attempted to tease out what makes a difference14,38 are a welcome addition to the literature, hinting as they do that particular subgroups of students may respond best to remediation.

 

이러한 결과는 놀랍지 않다. 첫째로, 오래 전부터 poor한 수행능력의 이유는 무수하다라는 것이 인식되어왔으며, 이는 poor performer가 균일한 집단이 아님을 보여준다. 둘째로 복잡한 인터벤션에서 나온 근거는 서로 다른 서브그룹간 서로 다른 효과가 있음을 보여주었다.

These findings are unsurprising. Firstly, it has long been recognised that reasons for poor perfor- mance are myriad; that is, poor performers are not a homogeneous group. Secondly, evidence from com- plex interventions in clinical areas has recently highlighted that these have different effects in different subgroups.61

 

가장 널리 활용되는 이론적 프레임워크는 대체로broadly 인지적이며, 자기조절, 메타인지, 성찰을 활용하고, 피드백을 주고 받는 것을 포함한다. 이는 매우 적절한데, weak 학습자와 strong 학습자는 조절 프로세스에서 질적 양적 차이가 있음을 교육 연구에서 보여주고 있다. 자기조절수준이 높은 학습자는 SRL 스킬이 낮은 학습자보다 학업적으로 더 성과가 높다.

The most widely used theoretical framework was broadly cognitive, using self-regulation, meta- cognition and reflection, and the giving and receivingof feedback.14,38 This seems very appropriate: educa-tion research has indicated quantitative and qualita- tive differences in regulation processes and activities between weak and strong learners (e.g. 62). Highly self-regulated learners are academically more suc- cessful than those students with low levels of skill in self-regulated learning (SRL) or those who lack regulation in their learning (e.g. 63).

 

의과대학생이 처음 의과대학에 들어왔을 때부터 자기조절을 잘 할것이라고 기대할 수 없다. 실제로, 탐색적 연구의 결과를 보면 successful한 의과대학생과 unsuccessful한 의과대학생이 SRL에 차이를 발견할 수 있다. 30여년의 교육 연구를 보면 SRL 테크닉에 대한 분명한 훈련이 효과가 있음을 보여줘왔다. '학습에 관한 학습' 과정을 의과대학 1학년 학생들에게 제공하는 것이 medical career의 초반 단계에 스스로의 SRL 접근법을 발견하게 해주고 변화를 할 수 있게 해줌으로서 미래의 underperformance의 가능성을 낮춰주는 효과적인 접근법이다.   

We cannot assume that students can self-regulate when they enter medical school: indeed, an exploratory study suggests that differences in SRL in successful and unsuccessful medical student learners are identifi-  able.64 Thirty years of education research has identi- fied that explicit training in SRL techniques is effective65,66 in terms of improving learning out- comes for students. We suggest that this framework could make important contributions to traditional medical training assessment frameworks that have been used to identify and remediate strugglers (see also67–69). The provision of ‘learning to learn’ courses for Year 1 medical students may provide an effective approach to helping students at an early stage of their medical careers to identify their SRL approach and make changes that might reduce their chances of future underperformance.70

 

대부분의 재교육에 대한 노력은 의과대학의 후반부에 있는 학생들이 대상이 된다. 그러나 조기 재교육 인터벤션은 많은 struggling student의 특징이기도 한 cycle of underperformance를 멈출 수 있는 잠재력을 지니고 있다. Struggling student는 낮은 자기효능감 신념과 학습에 부정적인 감정을 가지고 있어서 어려운 학습과제를 지속하고자 하는 동기부여에 영향을 준다. 이러한 학생들은 그들이 struggling student로 확인되는 이후 최대한 이른 시기에 성공을 경험할 필요가 있으며, 이를 통해서 스스로의 학습과 수행능력에 대한 통제 경험을 느껴볼 수 있게끔 해야 한다.

Most remediation efforts are targeted at learners in the latter years of medical school. However, early remediation interventions have the potential to stop the cycle of underperformance that is characteristic of many struggling students. Struggling students have low self-efficacy beliefs and negative feelings about learning that directly influence their motivation to persist with difficult learning tasks.71 These students need to experience success as soon as they are identified as struggling so that they can feel a sense of control over their learning and performance.

 

교육, 학습에 초점을 둔 변화와 학습의 실제 프로세스에 대한 평가를 포함하는 것이 효과가 있을 것이다. 그렇게 해야만이 underperformance에 대한 정확한 진단을 할 수 있고, 그것이 발생하는 이유를 찾아낼 수 있으며, '시험 잘보기 코칭'이 아닌 '학습을 위한 학습'에 초점을 둔 재교육을 위한 타당한 이론적 기반을 제공해줄 것이다.

changing the focus of teaching, learning and assessment to include assessment of the actual processes of learning would progress work in this area. Doing so should enable the accurate diagnosis of underperformance and the early identification of the reasons for its occurrence, as well as providing a sound theoretical basis for remediation that focuses on ‘learning to learn’ rather than ‘examination coaching’.

 

38 Winston KA, van der Vleuten CP, Scherpbier AJ. An investigation into the design and effectiveness of a mandatory cognitive skills programme for at-risk medical students. Med Teach 2010;32 (3):236–43.

64 Cleary TJ, Sandars J. Assessing self-regulatory processes during clinical skill performance: a pilot study. Med Teach 2011;33 (7):368–74.

67 Durning SJ, Cleary TJ, Sandars J, Hemmer P, Kokotailo P, Artino AR. Perspective: viewing ‘strugglers’ through a different lens: how a self-regulated learning per- spective can help medical educators with assessment and remediation. Acad Med 2011;86 (4):488–95.

68 Sandars J, Cleary TJ. Self-regulation theory: applica- tions to medical education: AMEE Guide No. 58. Med Teach 2011;33 (11):875–86.

69 White CB, Gruppen LD. Self-regulation learning in medical education. In: Swanwick T, ed. Understanding Medical Education. Chichester: Wiley-Blackwell 2010;271–82.

70 Sandars J. Pause 2 Learn: developing self-regulated learning. Med Educ 2010;44 (11):1122–3.

71 Artino AR Jr, Hemmer PA, Durning SJ. Using self-reg- ulated learning theory to understand the beliefs, emotions, and behaviours of struggling medical students. Acad Med 2011;86 (10 Suppl):35–8.

---

Med Educ. 2013 Mar;47(3):242-51. doi: 10.1111/medu.12052.

The remediation challenge: theoretical and methodological insights from a systematic review.

Cleland J1, Leggett H, Sandars J, Costa MJ, Patel R, Moffat M.

Author information

• 1Division of Medical and Dental Education, University of Aberdeen, Aberdeen, UK. jen.cleland@abdn.ac.uk

Abstract

OBJECTIVES:

Remediation is usually offered to medical students and doctors in training who underperform on written or clinical examinations. However, there is uncertainty and conflicting evidence about the effectiveness of remediation. The aim of this systematic review was to synthesise the available evidence to clarify how and why remediation interventions may have worked in order to progress knowledge on this topic.

METHODS:

The MEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ERIC (Educational Resources Information Centre), Web of Science and Scopus databases were searched for papers published from 1984 to April 2012, using the search terms 'remedial teaching', 'education', 'medical', 'undergraduate'/or 'clinical clerkship'/or 'internship and residency', 'at risk' and 'struggling'. Only studies that included an intervention, then provided retest data, and reported at least one outcome measure of satisfaction, knowledge, skills or effects on patients were eligible for inclusion. Studies of practising doctors were excluded. Data were abstracted independently in duplicate for all items. Coding differences were resolved through discussion.

RESULTS:

Thirty-one of 2113 studies met the review criteria. Most studies were published after 2000 (n=24, of which 12 were published from 2009 onwards), targeted medical students (n=22) and were designed to improve performance on an immediately subsequent examination (n=22). Control or comparison groups, conceptual frameworks, adequate sample sizes and long-term follow-up measures were rare. In studies that included long-term follow-up, improvements were not sustained. Intervention designs tended to be highly complex, but their design or reporting did not enable the identification of the active components of the remedial process.

CONCLUSIONS:

Most remediation interventions in medical education focus on improving performance to pass a re-sit of an examination or assessment and provide no insight into what types of extra support work, or how much extra teaching is critical, in terms of developing learning. More recent studies are generally of better quality. Rigorous approaches to developing and evaluating remediation interventions are required.

© Blackwell Publishing Ltd 2013.

Comment in

• Remediation for struggling learners: putting an end to 'more of the same'. [Med Educ. 2013]

PMID:

 

23398010

 

[PubMed - indexed for MEDLINE]