Blurprinting을 위한 12가지 팁(Med Teach, 2009)

Blurprinting을 위한 12가지 팁(Med Teach, 2009)

Twelve tips for blueprinting

SYLVAIN CODERRE, WAYNE WOLOSCHUK & KEVIN MCLAUGHLIN

Office of Undergraduate Medical Education, University of Calgary, Canada

도입

Introduction

타당도는 모든 평가에서 필요한 것이고, 지원자가 학습목표에서 설정한 역량의 수준에 맞는 최소한의 퍼포먼스를 달성했음을 의미한다. 전형적으로 학업성취의 측정과 관련된 타당도의 유형은 '내용타당도'이다. 내용타당도를 갖추었다는 것은 목표와 학습경험이 서로 부합하는 것이며, 이 pillars of education들 간의 '합치'는 평가 blueprint를 사용하여 향상될 수 있다.

Validity is a requirement of every evaluation and implies that candidates achieving the minimum performance level have acquired the level of competence set out in the learning objectives. Typically, the type of validity that relates to measurements of academic achievement is content validity (Hopkins 1998). Evaluation content is valid when it is congruent with the objectives and learning experiences, and congruence between these pillars of education can be facilitated by using an evaluation blueprint (Bordage et al. 1995; Bridge et al. 2003).

교육내용을 표로 만들라

Tip 1. Tabulate curricular content

 

열과 행으로 되어있다.

A blueprint template consists of a series of rows and columns. At the University of Calgary, teaching of the undergraduate curriculum is organized according to clinical presentations, so the rows in our blueprints contain the clinical presentations relevant to the course being blueprinted (Mandin et al. 1995). Column 1 in Table 1 shows the eighteen clinical presentations for the Renal Course at the University of Calgary. 

 

상대적 가중치를 준다

Tip 2. Provide relative weighting of curricular content

평가 문항의 수는 제한적이다.

Evaluations have a finite number of items,

그러나 어떤 내용이 중요한지를 정의하는 것은 어렵다. 오진시에 환자에게 영향impact가 큰 것 또는 질병예방이 중요한 것, 빈도 등을 기준으로 정의할 수 있다.

Content importance, however, is difficult to define. Attributes such as the potential harm to the patient from misdiagnosing a presentation (a measure of presentation ‘impact’), the potential for significant disease prevention (also a measure of presentation ‘impact’), and how frequently a presentation is encountered in clinical practice should be considered.

University of Calgary 에서는 다음의 세 가지를 가지고 판단한다.

At the University of Calgary we rate the impact and frequency of clinical presentations based on the criteria shown in Table 2.

모든 관련 그룹으로부터 의견을 모은다.

Tip 3. Sample opinion on weighting from all relevant groups

신뢰도는 표본의 size and breadth에 의해서 향상된다. 과목 책임자와 평가 담당자 외에도 교사, 학습자 등의 input을 얻어야 한다. 합의를 통해 가중치를 결정하는 것이다.

Reliability is improved by increasing sample size and breadth (Hopkins 1998). In addition to involving course chairs and evaluation coordinators, we solicit input from teachers and, if relevant, previous learners (McLaughlin et al. 2005a). That is, weighting of a content area is established through consensus.

각 내용영역에서의 문항 수 결정.

Tip 4. Decide on the number of items for each content area 

신뢰도는 변별도와 문항 숫자에 의해 영향을 받는다. 변별도가 0.3일 경우 0.8의 신뢰도를 위해서 50~60개 문항이 필요하고, 변별도가 0.2라면 100개가 필요하다. 100개를 넘어서는 신뢰도가 거의 향상되지 않는다.

Reliability of an evaluation is affected by both the number and discrimination of items. As a rough guide, if the average discrimination index of the items is 0.3, then approximately 50–60 items are needed to achieve reliability of 0.8. This number increases to 100 if the average item discrimination is 0.2. Reliability appears to plateau beyond 100 items (Hopkins 1998). 

각 내용영역에서 수행해야 할 task 정하기

Tip 5. Decide on the tasks for each content area

각 임상표현에 대해서 평가할 수 있는 다양한 task가 있다.

There are a variety of tasks that can be evaluated within any clinical cause presentation, such as

• diagnosing the underlying(including specific points of history and physical examination),

• interpreting or selecting investigations,

• decid-ing on management and/or prevention,

• demonstrating basic science knowledge, etc.

 

이들 task는 학습목표와 일관되어야 한다.

These tasks should be consistent with the learning objectives of the relevant course

 

 

이제 blueprint는 완성되었다. 다음은 valid content를 만드는 것이다.

The blueprint for content validity is now complete; the next challenge is to create the valid content. 

 

 

Blueprint 기반으로 평가 만들기

Tip 6. Create evaluations based on the blueprint

모든 평가는 blueprint와 부합해야 한다.

All evaluations used in the course – formative, summative and retake – should conform to the blueprint. 

문제은행 사용하기(또는 만들기)

Tip 7. Use (or create) an item bank

문제은행을 사용하는 것은 평가의 부담을 줄여준다. 문제은행이 존재하지 않으면, 장기적으로는 시간과 노력을 좀 들여서 만드는 편이 나으며, 과목이나 의과대학 간 공유할 수도 있다.

Using an item bank to match existing items to the blueprint reduces the burden of creating evaluations. If an item bank does not exist, the short-term investment of time and effort to create this pays off in the long run as items can then be shared between courses and even between medical schools. 

학습목표를 Revise

Tip 8. Revise learning objectives

 

blueprint는 과목의 모든 측면에 걸쳐서 가중치를 준다. 이 Weighting은 과목담당자가 과목의 학습목표를 생각해볼 수 있는 기회가 된다. 학습목표를 blueprint 기반으로 수정하는 것이 직관에 반할지는 모르나, 내용타당도를 달성하기 위해서는 목표의 숫자/교육 시수/평가 문항 수 등이 각 임상표현과 비례해야 한다.

 blueprint provides weighting for all aspects of a course. This weighting provides an opportunity for the course chair to reflect on the learning objectives. While it may appear counterintuitive to revise learning objectives based upon a blueprint weighting, to achieve content validity the number of objectives, hours our instruction, and number of evaluation items for each clinical presentation should be proportional.

학습경험을 revise

Tip 9. Revise learning experiences

 

Blueprint는 학습경험에 대해서 성찰해 볼 기회도 준다. 그러나 이것이 가중치와 교육시수 사이에 완벽한 선형관계가 있음을 의미하는 것은 아니다.

The weighting provided by the blueprint also offers an opportunity for reflection on learning experiences But this does not imply a perfect linear relationship between weighting and hours of instruction;

Blueprint를 교수들에게 배포하라

Tip 10. Distribute the blueprint to teachers

Blueprint는 과목책임자가 적절히 내용을 선태하는 것에 도움을 줄 뿐만 아니라, 교수들이 내용이 학습목표와 평가 두 가지 모두와 잘 부합하는 학습경험을 계획하는데도 도움이 된다.

The detail contained within a blueprint not only helps the course chair to select appropriate content area, but also helps teachers plan the learning experiences so that the content delivered is congruent with both the objectives and the evaluations.

내용 타당도를 모니터링하라

Tip 11. Monitor content validity

그러나 Blueprint를 발행하는 것 만으로 adoption되진 않는다. 내용타당도는 꾸준히 평가되고 모니터링 되어야 한다.

It cannot be assumed however, that publishing a blueprint inevitably leads to its adoption – content validity still needs to be evaluated and monitored.

Blueprint를 학습자에게 배포하라

Tip 12. Distribute the blueprint to learners

이상적으로, blueprint를 만들고 학습자들에게 배포하는 것은 course leader가 '운전대를 잡게' 해주며, 핵심 course material을 향해 학습이 이뤄지게 해준다.

Ideally, creating and providing a blueprint to learners ensures that course leaders are ‘grabbing hold of the steering wheel’ and driving learning towards what is felt to be core course material.

blueprint 는 학생들의 시험준비에도 중요한데, 학습목표를 reinforce시키고, 의도한 교육과정을 전달하게 해준다. 이전 연구에서 blueprint를 출판하는 것이 학생의 퍼포먼스를 향상시키지는 않았지만, 평가 프로세스의 공정함에 대한 인식을 크게 향상시켜주었다.

blueprint shown to be important in student examination preparation, reinforces the learning objectives and delivery of the intended curriculum (McLaughlin et al. 2005b). In a previous study, we found that blueprint publication did not improve student performance, but significantly increased the perception of fairness of the evaluation process (McLaughlin et al. 2005c).

 

---

Med Teach. 2009 Apr;31(4):322-4. doi: 10.1080/01421590802225770.

Twelve tips for blueprinting.

Coderre S1, Woloschuk W, McLaughlin K.

Author information

• 1Office of Undergraduate Medical Education, University of Calgary, Calgary, Alberta, Canada.

Abstract

BACKGROUND:

Content validity is a requirement of every evaluation and is achieved when the evaluation content is congruent with the learning objectives and the learning experiences. Congruence between these three pillars of education can be facilitated by blueprinting.

AIMS:

Here we describe an efficient process for creating a blueprint and explain how to use this tool to guide all aspects of course creation and evaluation.

CONCLUSIONS:

A well constructed blueprint is a valuable tool for medical educators. In addition to validating evaluation content, a blueprint can also be used to guide selection of curricular content and learning experiences.

PMID:

 

18937095

 

DOI:

 

10.1080/01421590802225770

[PubMed - indexed for MEDLINE]