북미의 의사면허시험: 외부 감사External audit의 가치는? (Med Teach, 2009)

북미의 의사면허시험: 외부 감사External audit의 가치는? (Med Teach, 2009)

Licensing examinations in North America: Is external audit valuable?

DONALD E. MELNICK

National Board of Medical Examiners, Philadelphia, USA

미국

In the United States, the United States Medical Licensing Examination (USMLE), sponsored, developed and adminis- tered by the Educational Commission for Foreign Medical Graduates, the Federation of State Medical Boards and the National Board of Medical Examiners, is required for entry into practice of all international medical graduates and US graduates holding the MD degree.

 

캐나다

In Canada, the Medical Council of Canada’s (MCC) qualifying examinations have the same function, leading to the Licentiate of the MCC, a credential recognized for licensure in Canada.

 

의무는 아니지만 ABMS, RCPSC, CFPC는 추가적인 audits of the competence를 제공한다.

While not mandatory for medical practice in the US, the national specialty certifying examinations provided by the members of the American Board of Medical Specialties, the Royal College of Physicians and Surgeons of Canada and the College of Family Physicians of Canada provide additional audits of the competence of doctors entering specialty practice.

미국과 캐나다의 면허시험은 명쾌한 설계규격에 따라 만들어진다. 이 설계규격에는 검사되어야 할 내용과 역량을 전문가 합의로 정의하여 만든 의료행위에 관한 descriptive data가 있다. Blueprint에 따라 출제하여 시험 형식에 따라, 시기time에 따라 일관성을 유지한다. 현재 미국과 캐나다 모두에서 시험에 포함되는 내용으로는..

Both US and Canadian licensing examinations are built to explicit design specifications. These specifications incorporate descriptive data about medical practice with expert consensus in defining the content and competencies to be tested. Careful adherence to examination blueprints assures consistency of examinations across test forms and across time. Some of the content categories currently included in both US and Canadian licensing exams are:

• understand and apply fundamental science,

• understand and apply clinical science,

• diagnosis,

• patient management,

• communication skills,

• history and physical examinations,

• critical appraisal,

• law and ethics and

• health promotion and maintenance.

북미의 면허시험은 점차 competency framework에 초점을 두고 있다.

The North American licensing examinations are increas- ingly focused on a broad competency framework.

일반적으로 인정받을 수 있는 competency framework를 사용함으로써 기존의 도구로 '평가가 용이한' 영역 뿐 아니라 다양한 중요한 역량을 평가할 수 있는 평가도구 개발을 자극한다.

Use of a generally accepted competency framework stimulates the development of assessment tools that assess the range of important competencies rather than just the domains that are easy to assess with existing tools.

이 국가적 프로그램들national programs에서는 모든 관련된 이해관계자들이 의료행위에 진입하기 위한 기준에 대한 합의를 개발하는데 포함시킨다. 설계규격과 정책, 시험 내용, 기준 등의 설정과정에 참여한다.

These national programs provide a vehicle for all relevant stakeholders to engage in development of consensus about standards for entry into practice. Stakeholders engaged in the USMLE include

• clinicians,

• academics,

• regulators,

• patients and students;

they are engaged in the process of developing design specifications and policies, test content and performance standards.

 

구성요소

USMLE has four examination compo- nents administered during medical school and at the point of licensure, and

the MCC examination has two components administered at the end of medical school and at the point of licensure.

 

IMG정책

In the US, international graduates must pass all components of USMLE; the first three components are required before entry into graduate medical education.

In Canada, international graduates complete a separate evaluating exam- ination to become eligible for the MCC qualifying examina- tions.

 

문항 유형

Both US and Canadian examinations include:

• enhanced multiple-choice questions (including items enriched with multimedia stimuli, low-fidelity simulation through sets of items around a single clinical case, multistep items requiring synthesis and application of knowledge to clinical problems);

• standardized-patient based assessments focusing on interpersonal, communication and clinical skills as well as professional behaviour; and

• assessment of medical problem solving and patient management using innovative test formats, such as the computer-based patient care simulation used in USMLE Step 3.

비용

USMLE administers about 141,000 tests annually; MCC administers about 10,000. USMLE occupies about 39h in four test sessions; MCC occupies about 10.5 h in two test sessions. The aggregate cost of USMLE to examinees is $2700; MCC aggregate cost is Canadian $2150.

미국과 캐나다 의료에 대한 가치는 아래의 여섯 가지이다.

Their value to American and Canadian medicine can be summarized in six key arguments:

• . 공통의 스탠다드의 평등 Equity of common standards. 

• . 외부 감사를 통한 투명성과 책무성 External audit providing transparency and accountability. 

• . 혁신을 위한 경계가 있는 환경 Providing a bounded environment for innovation. 

• . 근거-기반 교육과 규제를 위한 자료 제공 Providing data for evidence-based education and regulation. 

• . 근거-기반 노동력 유동성 Encouraging evidence-based workforce mobility. 

• . 양질의 평가 촉진 Fostering high quality assessment.

• 
  

공통의 국가시험시스템은 공통의 기준에 따르는 평등함을 제공한다. 국가시험이 도입되기 전에는 미국의 면허는 각 주의 권한이었다. 1964년 20개의 각 주별 시험을 검토한 결과, 시험의 내용과 타당도에 상당한 차이가 있다는 우려가 제기되었다. 또 다른 연구에서는 주별로 치러지는 시험의 합격률을 검토하였는데 3개의 주는 10년간 불합격자가 아무도 없었고, 6개주에서는 최소한 10년 중 8년 이상에서 불합격자가 없었다. 서로 다른 다양한 그룹이 '별도로, 하지만 동등하게' 평가될 수 있다는 정책주장은 (다른 나라나 문화에서와 마찬가지로) 미국에서도 완전히 실패한 것이었다. 단일한 국가시험은 의사들의 equity를 보장해주며, 교육적/지역적 배경을 비롯하여 '역량'의 결정에 대해 무관한 다른 여러 요인의 영향과 combating 해준다.

Common national assessment systems provide the equity inherent in common standards. Prior to the universal adoption of a national examination system in the US, licensure was based on assessments provided by each state licensing authority. A study assessing 20 of these examinations in 1964 documented high variability and concerns for the quality of test content and validity (Derbyshire 1965). Another study reviewed pass rates for state-based examinations from 1954 to 1964; three states had no failures in this 10-year period, and an additional six states had no failures for at least 8 of the 10 years studied (Miller 1964). Policies that assert that different groups can be treated ‘separately but equally’ have been utter failures in US race relations as in many other countries and cultures. A single national examination system assures equity for doctors, combating the influence of different perceptions of educational or regional background or other irrelevant factors on decisions about competence.

미국의 대통령이었던 로널드 레이건은 핵 비무장화에 대해서 '신뢰하지만 검증하라'라는 단순한 철학으로 접근하였다. 다수의 사회적 시스템은 '외부 감사external audit'의 가치가 정직성, 투명성, 신뢰에 있다고 인정한다. 의료전문직의 동기와 신뢰에 대한 공공의 회의감이 늘어나는 이 시대에, 국가면허시험은 공정하게 적용될 수 있는 단순하고 투명한 수단을 제공하여 환자들이 어떤 의사를 만나든 최소한의 역량 스탠다드를 보장할 수 있게 해준다. 주마다 지역마다 교육기관마다 서로 다른 복잡한 평가 시스템은 한 시스템에서 평가받은 의사가 다른 맥락에서도 동일하게 평가받을 수 있다는 공통의 assurance를 제공해주지 못한다.

US President Ronald Reagan approached nuclear disarma- ment with a simple philosophy: ‘trust but verify’. Many societal systems recognize the value of external audit in supporting honesty, transparency and trust. In an era of increasing public scepticism about the motives and trustworthiness of the medical profession, national licensing examinations equitably applied provide a simple, transparent means of assuring our patients that doctors have met minimum standards of competency. Complex systems of assessment that vary by state, region or educational institution fail the test of transparency, providing no common assurance that doctors assessed in one system have met the same standard as those assessed in another context.

플렉스너 보고서 시절부터 지금까지 MRCGP, MRCP and PACES 시험에서 의과대학간 systematical한 차이가 있음이 보고되어 왔으며, 여러 근거들은 일관되게 서로 다른 의과대학에서는 서로 다른 결과물이 나온다는 것을 보여준다.

From the Flexner report in the US in 1910 through recent studies documenting systematically different performance based on medical school attended on the MRCGP, MRCP and PACES examinations (Wakeford et al. 1993; McManus et al. 2008), evidence consistently demonstrates that different medical schools produce different results.

당연히, 국가시험이 교육 프로세스에서 핵심적 부분으로서의 'evaluation of progress'를 대체하지는 못한다. 그러나 external audit 없이는 교육 프로세스에 통합되어있는 평가에 대한 신뢰와 여러 기관 간 차이를 verify하지 못할 것이다.

Of course, national examinations are not a substitute for effective evaluation of progress as an integral part of the educational process. However, without external audit like that provided by a national assessment system, trust in assessment that is integrated into the educational process and differs from institution to institution will be impossible to verify.

잘 설계된 국가평가시스템은 boundary conditions, 즉 의료행위를 하는 개개인에게 기대되는 최소한의 요건을 설정해준다. 이 core expectation을 개인과 기관 수준에서 일관성을 만족시킬 수 있는 효과적인 도구를 제공한다. 환자는 어느 곳에 있든 그들을 진료하는 의사가 진료에 필요한 최소한의 지식/술기/행동을 마스터했기를 기대할 권리가 있다.

Well designed national assessment systems provide boundary conditions, establishing minimum expectations for individuals wishing to practice medicine. They offer an effective tool in assuring consistency in meeting those core expectations at the individual and institutional levels. Patients everywhere have the right to expect that any doctor they consult has demonstrated minimum levels of mastery of theknowledge, skills and behaviours necessary to practice.

2004년 미국(과 캐나다)에서 광범위한 이해관계자들이 임상술기와 관련된 역량의 평가를 국가시험에 포함시킬 것을 합의하에 지시하였다. 의학교육에서 임상술기를 가르치고 평가하는 것이 핵심적 요건core expectation이라는 오래된 합의에도 불구하고 (의과대학 인증기준에는 1990년대부터 포함되어 있었음), 임상술기에 대한 평가 도입은 2004년에야 이루어졌을 당시 1/3의 의과대학에서는 학생의 clinical skill을 평가할 공식적인 시스템을 갖추고 있지 않았다. USMLE에 도입되고 1년이 지났을 때 비록 접근법은 서로 달랐지만 거의 모든 학교에서 그러한 시스템을 갖추었다. 국가적 스탠다드가 존재함으로써 boundary condition이 설정되었으며, 모든 교육기관이 core expectation을 만족시킬 것이 권고되었고, 동시에 학교마다 매우 다양하고 혁신적 접근법을 허용하였다.
In 2004 in the US (a decade earlier in Canada), broad stakeholder consensus directed that the national examinations incorporate assessment of competencies related to clinical skills. Despite years of agreement that teaching and assessing clinical skills within medical education is a core expectation (as reflected through medical school accreditation standards since the early 1990s), at the time of the implementation of clinical skills assessment in USMLE in 2004, nearly one-third of medical schools did not have formal systems in place to assess their students’ clinical skills. Within 1 year of the USMLE clinical skills examination implementation, nearly all US schools had implemented such systems, although through very diverse approaches (Giluland et al. 2008). The presence of national standards sets boundary conditions that encourage all educa- tional institutions to meet core expectations while, at the same time, permitting highly variable and innovative approaches within the schools.

국가면허시험은 근거-기반 교육과 규제를 위한 자료를 제공한다. 여러 출판된 연구를 보면 국가시험의 수행능력과 임상에서의 수행능력이 정적 관계를 가진다.

National assessments for licensure provide data for evidence-based education and regulation. Several published studies show positive relationships between performance on national examinations and clinical performance (Tamblyn et al. 1998, 2002, 2007; Norcini et al. 2002; Papadakis et al. 2005; Holmboe et al. 2008).

• Tamblyn and her colleagues have demonstrated clear relationships between performance on examinations Canadian licensing and subsequent clinical performance in primary care over at least 10 years of practice.

• They have also demonstrated a relationship between assess- ment of clinical skills in a licensing examination and subsequent complaints to licensing authorities.

• Norcini and colleagues and Holmboe and colleagues have also shown positive relationships between scores on the internal medicine certifying and recertifying examinations and subsequent measures of clinical performance.

• Studies by Papadakis and colleagues, using the very crude outcome measure of disciplinary actions by licensing authorities, showed among significantly lower licensing examination scores disciplined physicians,

이 연구들은 지식과 술기가 부족한 의사를 가려내기 위한 시험의 활용을 지지하는 근거를 제공한다. 이러한 자료들을 모으는 것은 국가시험이 없이는 불가능할 것이다. 국가시험에 기반한 연구는 개개의 기관과 개별 의사들에게 중요한 benchmarking을 제공해주며, 특히 시험이 교육 프로세스의 기대 성과와 잘 align된 경우 더 그러하다. 

These studies provide evidence to support the use of examinations to identify those who may lack knowledge and skills to practice effectively. Aggregation of such data would not be possible without national assessment systems. Studies based on national examination data provide valuable benchmarking for individual institutions and indivi- dual doctors, particularly when the examinations are well aligned with the expected outcomes of the educational process.

국가시험은 의료인력의 유동성을 지원해준다. 인력의 유동성은 물론 국가시험 없이도 가능하지만 (유럽처럼) 국가시험은 근거-기반 portability를 제공해준다.

National examinations support mobility of the medical workforce. Of course, such mobility can occur without national examination programs, as is the case in Europe today; however, national examinations allow evidence-based port- ability of practice credentials.

마지막으로, 국가시험은 자원의 aggregation을 가능하게 해주는데 (인적자원과 경제적 자원), 이는 양질의 평가를 위해서 필요하다. 의사국가시험이 들어가는 비용이 높아 보이지만 aggregate cost는 수백 수천의 기관이 개별적으로 투입하는 비용의 총합보다 훨씬 적다.

Finally, national examinations allow the aggregation of resources, both human and fiscal, necessary for high quality assessment. While the national examination systems described here are expensive, their aggregate cost is much less than the combined cost of assessment in the hundreds or even thousands of institutions involved in education of doctors.

많은 poorly designed 평가는 신뢰성있는 정보를 제공해주지 못하고, 따라서 학습과 프로그램의 효과성 향상에 기여하는 바가 적다. 그러나 신뢰도가 갖춰지더라도 이는 필요조건이지 충분조건은 아니다. poorly designed된 경우, local이든 national이든, 학습자와 교수자의 행동을 distort할 수 있다.

Many poorly designed assessments do not provide reliable information and are, therefore, of little utility in guiding learning or program effectiveness. However, while reliability is requisite, it is not sufficient. Poorly designed assessment systems, whether local or national, may distort the behaviour of learners and teachers (Newble & Jaeger 1983).

적절한 지식과 전문성을 가지고 문제를 해결해야만 평가가 효과적일 것이다.

Assessment is an art and science apart from medicine, and it will be effective only when adequate knowledge and expertise are brought to bear on its challenges.

---

Med Teach. 2009 Mar;31(3):212-4.

Licensing examinations in North America: is external audit valuable?

Melnick DE1.

Author information

• 1National Board of Medical Examiners, Philadelphia 19104, USA. dmelnick@nbme.org

Abstract

The United States and Canada both have long-standing, highly developed national systems of assessment for medical-licensure based outside the institutions of medical education. This commentary reviews those programs and explores some of the reasons for their implementation and retention for nearly a century. The North American experience may be relevant to dialog about national or European assessments for medical practice.

PMID:

 

19811117

[PubMed - indexed for MEDLINE]