오픈북 시험과 클로즈드북 시험 비교(Systematic Review) (Acad Med, 2016) [출력완료]

오픈북 시험과 클로즈드북 시험 비교(Systematic Review) (Acad Med, 2016)

Comparing Open-Book and Closed-Book Examinations: A Systematic Review

Steven J. Durning, MD, PhD, Ting Dong, PhD, Temple Ratcliffe, MD, Lambert Schuwirth, MD, PhD, Anthony R. Artino Jr, PhD, John R. Boulet, PhD, and Kevin Eva, PhD

정보접근성이 쉬워지면서 CBE의 적절성에 대한 의문이 생긴다. 일부 학자는 시험은 외부 리소스를 찾고, 이해하고, 평가하고, 사용하는 능력을 평가해야한다고 주장합니다. OBE (Open-Book Examination)의 지지자는 OBE가 실전 연습에보다 정통하며 성공은 "단순 암기"가 아니라고 주장합니다. 1-3

This easy access to information raises fundamental questions about the adequacy of closed-book examination (CBE) practices commonly used by the health professions. Some scholars argue that any examination of relevance must assess the examinee’s ability to find, understand, evaluate, and use external resources. Such proponents of the open-book examination (OBE) argue that OBEs are more authentic to real-world practice and that success is not about “rote memorization.”1–3 

CBE를 지지하는 학자들은 전문가의 성과는 풍부하고 잘 조직화된 내용지식과 밀접한 관련이 있음을 발견 한 문헌을 인용합니다. 예를 들어 연구에 따르면 CBE에 대한 높은 수행 능력은 보다 나은 practice outcome과 관련이 있음을 발견했습니다 .5,6 또한 많은 경우 정보를 찾아볼 수 있는 능력은 (여러 환경적 요인 등으로) 제한된다

Scholars defending CBEs cite literature that has consistently found expert performance to be closely tied to rich, well-organized content knowledge of a subject. For example, studies have found that high performance on CBEs is associated with better practice outcomes.5,6 In many situations a physician’s ability to look up unknown information is restricted

To inform this issue, which affects the examination of physicians across the continuum of their careers, we conducted a systematic review of the literature comparing the two assessment strategies.

단순히 의사의 손끝에 더 많은 정보를 제공하는 것만으로는 치료가 개선되지 않을 수 있다. 왜냐하면 의사는 어떤 것을 찾아봐야 할지에 대한 지식이 있어야 하고, 새로운 정보를 기존의 경험과 통합시킬 수 있어야 하기 때문이다. 따라서 정보 기술에만 의존하게되면 인지 부하 (즉, 정신적 노력)가 악영향을받을 수 있고 정보 학습 및 critical appraisal 능력이 저하되며 궁극적으로 환자의 건강을 해칠 수 있습니다 .7

Merely putting more information at a physician’s fingertips is, therefore, not likely to result in improved care because the physician needs knowledge to guide his or her search and to integrate new information with previous experience. Thus, reliance on information technology could detrimentally increase cognitive load (i.e., mental effort), decrease learning and critical appraisal of information, and ultimately harm patient care.7 

방법

Method

 

범위 Scoping search

Systematic review

We followed PRISMA Guidelines8 and guidelines provided in the medical education literature.9 We limited our search to full-length, published, peer- reviewed, English-language journal articles involving learners in either descriptive reports or educational interventions, using any study design related to our research questions. We further limited the papers reviewed to those that empirically compared (either directly or indirectly) OBEs and CBEs.

We used a data collection form (Supplemental Digital Appendix 2, http:// links.lww.com/ACADMED/A310) to rate each article. This form was constructed based on the findings of our scoping review and refined through conference calls among the authors.

We structured the outcome categories according to the themes that were generated from our scoping review. We report them here in the sequence in which they would occur in the testing process:

• (1) examination preparation,

• (2) test anxiety,

• (3) exam performance,

• (4) psychometrics and logistics,

• (5) testing effects, and

• (6) public perception.

Any article could have multiple outcomes and was reviewed for relevant themes by two of the study authors.

The quality of each manuscript was examined by addressing the extent to which the research found was fit for purpose. This was done by having each reviewer code the manuscript for the presence of explicit research questions, hypotheses, conceptual and/or theoretical frameworks, and by recording additional quality judgments. Reviewers used a five- point rating scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree) to assess four domains:

• trustworthiness of findings,

• study rigor,

• implementation of study findings, and

• appropriateness of data analysis.

 

These latter judgments were made in relation to the degree to which each study effectively addressed a research question comparing the relative benefits of OBEs versus CBEs.

 

결과

Results

The frequency with which outcomes were identified was as follows:

• (1) exam preparation (n = 20; 54%);

• (2) test anxiety (n = 14; 38%);

• (3) exam performance (n = 30; 81%);

• (4) psycho- metrics and logistics (n = 5; 14%);

• (5) testing effects (n = 13; 35%); and

• (6) public perception (n = 5; 14%).

연구 퀄리티

Study quality

Overall, the quality of the articles included in our review was deemed to be adequate for our purpose. Explicit research questions were presented in 31 articles (84%), hypotheses were stated in 14 (38%), and hypotheses were justified in 10 (27%). Conceptual and/or theoretical frameworks were described in 7 articles (19%).

연구 맥락

Study context

• Thirty-four investigations (92%) were single-institution studies.

• Nearly half were performed in the United States (n = 18; 49%).

• Other locations included the Netherlands (n = 5; 14%), the United Kingdom (n = 4; 11%), Greece (n = 3; 8%), and Australia (n = 2; 5%), and 1 study (3%) was included from each of the following locations: Canada, Denmark, Norway, Africa, and Israel.

대부분 대학생

The majority of studies pertained to college-level students (n = 24; 65%); 2 studies investigated high school students (5%); 8 investigated medical students (22%; 2 of these were multi- institutional); 2 investigated other postcollege instructional settings (5%); and 1 study (3%) included practicing physicians.

 

대부분 중-부담 시험

For the majority, the stakes of the examination were rated as medium (n = 21; 57%) in that the assessments were generally end-of- course examinations. Two (5%) were considered high-stakes, being equivalent to national licensing examinations. Few studies included a formal incentive (e.g., extra credit or a small payment) (n = 6; 16%) to participants beyond earning a course grade.

 

시험 준비

Exam preparation

종합해보면, 시험 준비 시간은 CBE에서 더 길었다.

In terms of preparation time, findings were inconsistent across studies, but in sum appear to favor CBEs.

시험 준비 시간이 길다는 것은 (학습이 향상되었음을 보여주는 대리지표라기보단) 사전에 학습자료와의 engagement가 불충분했다는 의미일 수 있다.

Of note, an increase in preparation time could indicate insufficient prior engagement with the material rather than being a proxy for improved learning and performance.15

OBE나 CBE에서 학습전략이 달라지지 않았다.

Reviewing the articles examining preparation strategy revealed that students did not change study tactics for OBEs versus CBEs,16,17 and no correlation between test format and deep versus surface learning approaches was found.17

학생이 어떻게 시험준비를 하는지는 CBE나 OBE나 차이가 별로 없다. 차이가 있었을 경우에는, 참가자들은 CBE일 때 더 공부하였다.
Thus, research exploring exam preparation was equivocal with respect to whether students prepare differently (or at greater length) for CBEs or OBEs. When differences did exist, they tended to show that participants studied more when they expected a CBE.

시험 불안

Test anxiety

감정은 인지 능력에 영향을 미친다. 부정적인 감정은 한때 성과에 독점적으로 해로운 영향만을 미쳤다고 생각되었지만 감정의 현대 이론은 그러한 가정이 지나치게 단순하다는 것을 암시한다. 예를 들어, 불안과 같은 부정적인 감정은 실제로 학생이 CBE는 OBE를 준비중인 스트레스를받지 않는 학생과 비교했을 때 뛰어난 성능을 나타낼 수 있습니다. 그럼에도 불구하고 시험 불안감을 줄이는 것이 종종 OBE를 고려한 첫 번째 동기라고보고됩니다. 그러나 우리의 연구 결과에 따르면, 불안 효과는 일반적으로 연구의 주요 목적 (부록 1 참조)과 관련하여 2 차적 문제로 조사되었으며, 감정을 평가 한 모든 연구에는 이론적 근거가 부족한 것으로 나타났습니다.

Emotions affect cognitive performance.44 Although negative emotions were once thought to have exclusively deleterious effects on performance, contemporary theories of emotion suggest that such an assumption is overly simplistic.45 For example, a negative emotion like anxiety might actually motivate a student to study for a CBE, which could result in superior performance when compared with an unstressed student preparing for an OBE. Regardless, reducing test anxiety is often reported to be a primary motivation for considering OBEs. Our findings indicate, however, that anxiety effects were typically examined as a secondary issue relative to a study’s primary purpose (see Appendix 1), and all studies that assessed emotions lacked a theoretical grounding.

 

학생들이 OBE 또는 부분적인 OBE (즉, 학생들이 원하는 자료에 액세스하지 않고 "치트 시트"와 같은 준비된 자료를 가져올 수있는 시험)가 자신의 불안을 줄이는 효과가 있다는 것을 과대 평가할 수 있다는 증거가 있습니다. 여러 연구에 의하면 학생들은 덜 우울한 태도로 OBE를 연관 시키지만, 소수의 학생 만이 실제로 불안감을 덜보고한다고 제안하고있다 .24,28 더 어려운 문제가 출제될 것이라는 생각은 학생들에게 불안감을 유발할 수 있습니다 .19

Evidence suggests that students may overestimate the effect that OBEs or partial OBEs (i.e., exams in which students can bring some prepared material like a “cheat sheet” rather than having access to any desired material) have on reducing their anxiety. Several studies suggest that students associate OBEs with less anxiety,16,27,28 but only a minority of students actually report lower anxiety.24,28 It has been suggested that certain aspects of OBEs, such as the belief that examiners will choose questions of greater difficulty, can be anxiety provoking for students.19

 

종합하면, 이러한 결과는 OBE가 불안을 ​​줄이는데 미치는 영향을 과대 평가할 수 있으며 잠재적으로 성적을 향상시킬 수 있음을 시사합니다. 불안 효과를 조사하기위한 방법과 분석에 대한 보고가 불완전하지만, 이러한 영향은 현존하는 연구의 afterthought로 탐구되며, 이론적 인 근거가 부족합니다.

On balance, these findings suggest that students may overestimate the impact that OBEs have on reducing their anxiety and, by extension, on potentially improving their performance. Not only was the reporting of methods and analyses for examining anxiety effects incomplete, but these effects are often explored as an afterthought in extant studies, and they lacked theoretical grounding.

시험 수행능력

Exam performance

가장 일반적인 결과는 학습자의 OBE 대 CBE 성취도 (부록 1)와 비교 한 시험 수행이었다. 직관적으로, 수험생이 책을 찾아볼 수 있기 때문에 OBE 성적이 더 좋을 것이라고 기대할 수 있습니다. CBE를 반대하는 사람들은 OBE 형식이 본질적으로 난이도를 낮추는 것은 아니지만 대신 시험에서 지식을 적용 할 수있는 능력에 대한 질문에 초점을 맞추도록 (즉, 간단히 책을 찾아서는 알 수 없는 것을 테스트하는) 하는 장점이 있음을 강조한다.
The most common outcome explored was examination performance, defined as comparing learners’ achievement on OBEs versus achievement on CBEs (Appendix 1). Intuitively, one might expect that examinees would perform better on OBEs because they have the capacity to look up answers. Opponents suggest that the OBE format does not inherently lessen difficulty but, instead, frees the examiner to focus questions on the test taker’s ability to apply knowledge (i.e., testing what cannot simply be “looked up”),

시험 성과를 결과로 고려할 때 두 가지 주의 사항이 주목할 만하다.

• (1) 대부분의 연구에서 학생들은 OBE에 대한 경험이 거의 없었습니다.

• (2) 시험 성적은 문제의 난이도 등에 따라 달라지므로 성과로서 평가하기에 어려운 부분이 있으며, 다만 OBE의 일부 지지자는 강사가 다른 스타일 또는 다른 포커스의 문항을 출제할 수 있는 점을 강조한다.
  

Two caveats are noteworthy when considering exam performance as an outcome: (1) In most studies, students had little to no experience with OBEs—only one study21 that addressed examination performance reported that students had prior OBE experience; and (2) exam performance is a challenging outcome to study because the difficulty of an exam depends on the questions asked, and some proponents of OBE argue that its main advantage is enabling instructors to pose questions with a different style or focus.

대부분의 시험은 MCQ 형식 이었지만 일부는 에세이 및 / 또는 짧은 대답이었습니다 (부록 1). 전형적으로, 수험자의 성과에서 중요한 차이는 발견되지 않았고, CBE (30,34,38)에서 성과가 더 좋았다 (부록 1).

The majority of the examinations were MCQ format, but some were also essay and/or short answer (Appendix 1). Typically, no significant difference in examinee performance was found,30,34,38 or performance was better on CBEs (Appendix 1).

결론적으로 CBE에서 시험성적이 더 높은 것처럼 보입니다. 그러나 OBE에 대한 경험이 비교적 적은 것과 두 시험에서 나타나는 시험 준비의 차이는 학생들에게 OBE에 대해 더 알려주거나, 연습시험을 제공하여 OBE에서의 학업성취를 향상시킬 수 있음을 제시한다. 학생들이 OBE에서 잘 하기 위해서는 무엇이 필요한지에 대한 right expectation을 가져야 한다고 언급했다.

In sum, studies comparing exam performance appear to favor CBEs. However, the combination of relatively little experience with OBEs and the differences in exam preparation noted in several investigations highlighted in this section leave open the possibility that OBE performance could be improved through instructing students about OBEs or providing practice tests. On this point, three sets of authors indicated that students need to have the right expectation for what it takes to do well on OBEs.19,21,24

Psychometrics 과 로지스틱스

Psychometrics and logistics

연구에 따르면 일반적으로 검사의 validity는 형식보다 내용에 따라 결정됩니다 .46-48
Research has generally shown that the validity of a test is determined more by the content of the questions included than by the examination format.46–48

실제로 문항 수를 일정하게 유지하면서 테스트 형식에 대한 안정성을 비교하는 것이 현실적이지 않을 수 있습니다. 시험을 치는 데 필요한 시간에 미치는 영향에 대해 CBE와 OBE를 비교 한 3 가지 연구에서 OBE를 완료하는 데 10 % ~ 60 % 더 오래 걸리는 것으로 나타났습니다 .10,30,32 따라서 시험시간의 양을 제어하는 ​​경우, OBE 형식에서의 문항이 더 줄기때문에 CBE 형식 시험의 신뢰도가 높아질 수 있습니다.

In practice, it may not be realistic to compare reliability across test formats while keeping the number of items constant. Three studies that compared CBEs with OBEs with respect to their influence on the time required to take the test found that students took 10% to 60% longer to complete OBEs.10,30,32 Thus, if one controls for amount of testing time, it is likely that fewer questions would be asked in OBE format, and, hence, the reliability of the equivalent CBE-formatted exam can be anticipated to be higher.

시험 효과

Testing effects

 

Testing effect는 시험을 본느 것 자체로 후속 성과가 향상되는 것을 말한다. 이러한 향상은 간접적인 방법 (예 : 공부하도록 유도하는 등) 또는 직접적인 방식으로 나타난다(참가자가 단순히 공부만 할 때보다 시험을 보는 과정에서 자료가 더 memorable하게 됨). study group보다 tested group이 성적이 더 좋은 걸 보여줘서 확인가능하다. 이 testing effect는 여러 분야에서 확인되었습니다.

Testing effects occur when taking an exam improves subsequent performance. Such benefits can arise in indirect ways (e.g., being prompted to study) or from direct effects of the material becoming more memorable when participants are tested on it than when they simply study for a test.49 The testing effect is demonstrated when the tested group outperforms the study group on a subsequent outcome exam. This testing effect (test-enhanced learning) has been well documented in multiple fields.50

 

CBE의 지지자들은 학습이란 기억의 능동적 구성이라고 주장하며, 문제에 답하기 위해서 외부 자료에 의존할 때는 발생할 가능성이 낮다고 주장한다. OBE 지지자들은 OBE에서는 단순암기가 강조되지 않기 때문에 OBE가 지식을 적용하는 능력을 향상시킬 수 있다고 주장합니다.

Proponents of CBE argue that learning requires active construction of memory that is less likely to occur when one relies on external resources to answer test questions. OBE proponents argue that OBEs may enhance the ability to apply knowledge because rote memorization is not emphasized.

OBE와 CBE 모두에서 시험 효과가 나타난다. OBEs와 CBEs를 비교 한 4 개의 연구는 testing effect가 두 방식에서 거의 동일 함을 보여 주었다.

Both OBE and CBE demonstrate testing effects (Appendix 2). Four studies comparing OBEs and CBEs demonstrated testing effects that were roughly equivalent10,13,31,37 (Appendix 2).

• The testing effect of CBEs was superior in one study.12 These researchers demonstrated that during a summative CBE participants performed worse on material covered by an OBE intervention relative to a CBE intervention.12

Testing effects는 시험 형식과 관계 없이 나타난다. 이것은 학생들의 collective self-perceptions 는 testing effect에 대한 연구결과와는 반대로 나타난다는 기존 연구 결과와도 부합하는 것이다. 학생들은 OBE든 CBE든 시험을 보는 것보다 공부만 하는 것studying alone이 더 효과적인 준비라고 느꼈다.

Testing effects were observed regardless of examination format. Consistent with prior studies, students’ collective self-perceptions ran counter to the empirical finding that testing effects occur regardless of test format; students felt that studying alone was more effective preparation than taking either an OBE or CBE.31

공공의 인식

Public perception

연구에 따르면 학습자는 CBE보다 OBE에 대해 긍정적인 인식을 가지고 있음을 보여줍니다. 반면에 학생들은 OBE 문제가 더 어렵고 OBE 형식에 대한 추가 연습이나 훈련이 필요하다고도 의견을 말했습니다 .17

Studies suggest that learners have a more positive perception of OBEs over CBEs.2,17,19,22 On the other hand, students also commented that OBE questions were more difficult and that they desired additional practice or training for the OBE format.17

교사들은 종종 OBE도입에 반대한다. 교사들은 OBE를 준비하는 과정에서 resource가 더 필요하다는 것, 그리고 OBE 준비에 요구되는 추가 시간에 대해 우려했다 .2,22

Teachers’ views often challenged the implementation of OBEs.17,23 Teachers expressed concerns over the increased resources associated with preparing OBEs, as well as the perceived additional time required for learners to take OBEs.2,22

 

고찰

Discussion

연구 분야 (예 : 의학 교육 대 교육)와 콘텐츠 수준 (예 : 대학생 대 대학원생))에 따라 OBE와 CBE 사이의 체계적인 차이점은 없었다. 연구를 하기 전에는 결과는 모호했다. 테스트 형식이 outcome에 영향을 미친다면, 사람들이 CBE를 볼 때에 많이 준비한다는 주장이 옳아야 한다. 이것은 학생들이 OBE를 볼 때 불안이 줄어들 것으로 기대하기 때문인데, 실제로 불안이 줄어드는 것으로 보이지는 않는다.

The studies we reviewed were generally of adequate quality for the questions addressed, and we did not identify any systematic differences in the use of OBE versus CBE by the field studied (e.g., medical education versus education versus other) or level of content (e.g., graduate versus undergraduate student). Prior to the examination, findings were equivocal; if test format does affect outcomes, it favors the argument that people prepare more for CBEs. This may be driven by the finding that students anticipate lessened anxiety with OBEs even though this does not appear to translate to actual experiences of lessened anxiety.

 

학생들은 OBE 시험을 보는데 데 더 오랜 시간이 걸리며, 시험 시간이 일정하다면, 이것은 신뢰도에 영향을 줄 수 있있고, 또는 동일한 신뢰도의 시험에 필요한 시간에도 영향을 줄 수 있다. 시험 성과에 대해서는 CBE에서 더 성과가 높았는데, 특히 학습자가 OBE보다 CBE를 준비하는 데 더 많은 시간을 할 경우 그러하였다. CBEs와 OBEs의 postexamination 결과와 관련하여, 우리는 testing effect 또는 public perception의 차이에 대한 확실한 증거를 찾지 못했다. 즉, 사람에 따라서는 "만약 인터넷에서 찾아봐야 뭘 알수 있는 사람이라면, 어떻게 전문가라 부를 수 있나?"라고 우려하는 환자가 있을 수 있다.

During the examination, examinees appear to take longer to complete OBEs, which could either influence the test’s reliability, if testing time is kept constant, or influence the length of time that must be offered to candidates to complete an equally reliable exam. Studies addressing examination performance favored CBEs, particularly when learners reported spending more time preparing for CBEs than for OBEs. With respect to postexamination outcomes of CBEs and OBEs, we did not find robust evidence for differences in testing effects or public perception. That said, one might imagine concerned patients who wonder, “How can you be an expert if you need to look things up on the Internet?”51

 

어떤 유형의 시험을 사용하는가는 학습성과나 수행능력성과에 따라서 결정되기보다는 로지스틱한 한계에 의해서 결정될 수 있으며, 또한 실제 진료상황에서는 어떻게 하게되는지에 대한 authentical representation을 위해서 결정될 수도 있다. OBE와 CBE 모두 testing effect가 있었기 때문에, 그리고 학생들의 testing effect에 대한 인식은 연구결과와 정반대로 나오기 때문에, 관련된 질문은 'testing effect를 최대화하기 위해서는 얼마나 자주 시험을 봐야 하는가'이다.

The type of examination used might need to be based less on learning and performance outcomes and more on logistical limitations, as well as the desire to authentically represent what individuals do in practice. Given that we found evidence of the testing effect under both OBE and CBE conditions, and that participants’ perceptions of testing effects run counter to empirical findings, a related question is how often an individual should be examined to maximize testing effects.

고부담시허에서 시험기간 동안 인터넷을 사용하게 하는 것은 시험보안에 위협이 된다. 동시에 제한된 수의 웹-기반 외부자료를 정해주는 것은 authenticity를 떨어뜨려서, 선택한 tool에 덜 익숙한 응시생에 불이익을 줄 수 있고, 시험을 치르는 동안 technical difficulties가 발생하면 공정성에도 영향을 준다. 또 다른 feasibility 문제로는 웹-기반 resource를 허용하는데 필요한 비용이며, OBE에서 동일한 신뢰도를 얻기 위에 필요한 추가적 비용이다.

It is challenging for high-stakes testing organizations that value test security to allow learners to have unrestricted access to the Internet during an exam.53 At the same time, choosing a limited number of Web-based external resources erodes authenticity, could disadvantage examinees who are less familiar with the chosen tools, and potentially affects fairness if technical difficulties arise during an examination. Additional feasibility questions include the cost of allowing Web-based resource access and the additional time required to achieve the same reliability with OBE relative to CBE. Issues such as cost and fairness have not been addressed in prior investigations.

authenticity와 관련해서, 지금까지는 '고부담'시험에 대한 연구가 거의 없었다. 정보검색능력이 의사에게 중요한 능력이라는 것에 대해서는 이론의 여지가 별로 없으나, OBE가 CBE보다 더 쉽다는 인식이 있을 수 있다. ABO의 발췌문

In terms of authenticity, the studies conducted to date have rarely looked at “high-stakes” assessment. Although there is good reason to argue that a physician’s ability to find information is an important skill to maintain, there can be a perception that OBEs are easier than CBEs. Although studies are lacking, an excerpt from the American Board of Ophthalmology regarding changes to their recertification examination captures the sentiment of many:

The decision to change from an open- book, take-home examination to a closed-book, computerized proctored examination was based primarily on the recognition of the value of the certificate within the public domain … state medical licensing boards are increasingly asking for a proctored(감독받는) examination.54

proctored exam을 선호하는 것이 OBE가 덜 엄격하고 validity가 낮다는 인식을 나타낸다.

We believe this preference is indicative of the perception that OBEs are perhaps less rigorous and/or less valid than a proctored examination.

OBE와 CBE는 서로의 장단점을 보완하기 때문에 부분적으로 평가 프로그램에 기여할 수 있습니다. OBE는 CBE의 대안으로 생각되어서는 안되지만, OBE의 가치는 CBE가 측정하는 것 이상에 있습니다. 예를 들어, 인터넷에서 정보를 찾는 "기술"을 평가하는 것은 CBE를 달성하기 어려울 것이다. 따라서 전략은 신뢰도를 손상시키지 않으면서 이러한 다양한 "기술"을 탐구하기 위해 OBE와 CBE를 결합 할 수 있습니다. 더욱이, testing effect와 관련하여 몇 번의 시험을 봐야 가장 적절한지의 문제는 아직 최적화되지 않았습니다. 의무적이지만 ungraded OBE는 학습자 불안을 크게 증가시키지 않으면서도 testing effect를 유도할 수 있다. 많은 인증 기관이 시행하는대로 매 10 년마다 한 번씩 시험을 치르는 것으로는 시험의 교육적 효과를 극대화하거나, 지속적인 전문직 개발의 습관 유도할 수 없다. 또한 OBE 항목을 부분적으로 포함시킴으로써 authenticity를 향상시킬 수 있으며, '찾아봐야만 안다'라는 식의 낙인stigma을 줄일 수 있다.

OBEs and CBEs can contribute to an assessment program in part because of their complementary pros and cons. OBEs should not be thought of as an alternative to CBEs, but their value may be in expanding beyond what is measured by CBEs. For example, exploring the “skill” of looking up information on the Internet seems unlikely to be accomplished through CBE. A strategy, therefore, could be coupling OBEs with CBEs to explore these different “skills” without compromising reliability. Furthermore, testing effects are not currently being optimized given the infrequency of examinations. A series of mandatory but ungraded OBEs might help to improve aspects of these processes, such as capitalizing on the testing effect without dramatically increasing learner anxiety. One examination each decade, as is practiced by many certifying bodies, is unlikely to maximize the educational impact of testing or induce habits of continuous professional development. Further, by including some OBE items, the opportunity for improving authenticity and reducing the stigma with the need to look things up could be leveraged.

Conclusions

---

Acad Med. 2016 Apr;91(4):583-99. doi: 10.1097/ACM.0000000000000977.

Comparing Open-Book and Closed-Book Examinations: A Systematic Review.

Durning SJ1, Dong T, Ratcliffe T, Schuwirth L, Artino AR Jr, Boulet JR, Eva K.

Author information

• 1S.J. Durning is professor of medicine and pathology, Uniformed Services University of the Health Sciences, Bethesda, Maryland. T. Dong is assistant professor of medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland. T. Ratcliffe is assistant professor of medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland. L. Schuwirth is professor of medical education, Flinders University, Bedford Park, South Australia, Australia, and professor for innovative assessment, Maastricht University, Maastricht, the Netherlands. A.R. Artino Jr is professor of medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland. J.R. Boulet is vice president of research and evaluation, Foundation for Advancement of International Medical Education and Research, Philadelphia, Pennsylvania. K. Eva is professor of medicine, University of British Columbia, Vancouver, British Columbia, Canada.

Abstract

PURPOSE:

To compare the relative utility of open-book examinations (OBEs) and closed-book examinations (CBEs) given the rapid expansion and accessibility of knowledge.

METHOD:

A systematic review of peer-reviewed articles retrieved from MEDLINE, ERIC, Embase, and PsycINFO (through June 2013). In 2013-2014, articles that met inclusion criteria were reviewed by at least two investigators and coded for six outcome categories: (1) examination preparation, (2) test anxiety, (3) exam performance, (4) psychometrics and logistics, (5) testing effects, and (6) public perception.

RESULTS:

From 4,192 identified studies, 37 were included. The level of learner and subject studied varied. The frequency of each outcome category was as follows: (1) exam preparation (n = 20; 54%); (2) test anxiety (n = 14; 38%); (3) exam performance (n = 30; 81%); (4) psychometrics and logistics (n = 5; 14%); (5) testing effects (n = 24; 65%); and (6) public perception (n = 5; 14%). Preexamination outcome findings were equivocal, but students may prepare more extensively for CBEs. For during-examination outcomes, examinees appear to take longer to complete OBEs. Studies addressing examination performance favored CBE, particularly when preparation for CBE was greater than for OBE. Postexamination outcomes suggest little difference in testing effects or public perception.

CONCLUSIONS:

Given the data available, there does not appear to be sufficient evidence for exclusively using CBE or OBE. As such, a combined approach could become a more significant part of testing protocols as licensing bodies seek ways to assess competencies other than the maintenance of medical knowledge.

PMID:

 

26535862

 

DOI:

 

10.1097/ACM.0000000000000977

[PubMed - indexed for MEDLINE]