우리가 언제 무엇을 가르치고 있지?

BRIEF REPORT: ‘‘Where Do We Teach What?’’

Finding Broad Concepts In The Medical School Curriculum

Joshua C. Denny, MD,1 Jeffrey D. Smithers, MD,2 Brian Armstrong, BS,3 Anderson Spickard III, MD, MS4

1Department of Medicine, Vanderbilt School of Medicine, Nashville, TN, USA; 2Department of Medicine and Pediatrics, Good Samaritan Regional Medical Center, Phoenix, AZ, USA; 3Vanderbilt School of Medicine, Nashville, TN, USA; 4Department of Medicine, Department of Biomedical Informatics, Vanderbilt School of Medicine, Nashville, TN, USA.

BACKGROUND: Often, medical educators and students do not know where important concepts are taught and learned in medical school. Manual efforts to identify and track concepts covered across the curriculum are inaccurate and resource intensive.

OBJECTIVE: To test the ability of a web-based application called KnowledgeMap (KM) to automatically locate where broad biomedical concepts are covered in lecture documents in the Vanderbilt School of Medicine.

METHODS: In 2003, the authors derived a gold standard set of curriculum documents by ranking 383 lecture documents as high, medium, or low relevance in their coverage of 4 broad biomedical concepts: genetics, women’s health, dermatology, and radiology. We compared the gold standard rankings to KM, an automated tool that generates a variable number of subconcepts for each broad concept to calculate a relevance score for each document. Receiver operating characteristic (ROC) curves and area-under-the-curve were derived for each ranking using varying relevance score cutoffs.

RESULTS: Receiver operating characteristic curve areas were acceptably high for each broad concept (range 0.74 to 0.98). At relevance scores that optimized sensitivity and specificity, 78% to 100% of highly relevant documents were identified. The best results were obtained with the application of 63 to 1437 subconcepts for a given broad concept. The search time was fast.

CONCLUSIONS: The KM tool capably and automatically locates the detailed coverage of broad concepts across medical school documents in real time. Use of KM or similar tools may prove useful for other medical schools to identify broad concepts in their curricula.

---

"우리가 유전학을 언제 배우고 있더라?" 학장도, 교수도, 학생도 이러한 질문을 한다. 이런 질문에 대해 답을 하려고 무수한 전화와 이메일이 오간다.

‘‘Where do we currently teach genetics?’’ is a typical question asked in medical schools. 

• A dean may pose this question in preparation of the school’s report to the Liaison Committee on Medical Education (LCME). 
• A faculty member may ask this question to prepare to teach a new genetics lecture added to the medicine clerkship. 
• A medical student clerk may ask this question to prepare for her postcall bedside presentation of her patient with complications of an inherited disease. 

Questions about where concepts and topics are covered in the curriculum are typically answered through lengthy collaborative meetings and numerous emails and phone calls. These labor- and communication-intensive approaches may not be sufficiently detailed and sustainable to yield accurate, up-to-date answers.

교육자들은 교육과정의 내용을 관리하고, 추적하고, 조율하기 위헤서 웹을 사용하기 시작했다.

Educators have turned to the web to display medical school information in order to improve the ability to manage, track, and coordinate curricular content. 

Some medical schools support a full text electronic curriculum so that teachers and learners have quick access to the current curriculum. 1,2 

Other medical schools take a step further by placing electronic curricular documents and images into a database. 3,4 

KnowledgeMap(KM)이란 프로그램을 개발했다.

Each document is labeled with a manually derived set of keywords or titles to allow for searching and sharing of curricular content. Improvements on these efforts would obviate the need for manual identification and entry of selected concepts that describe a teaching session, and yet grant the ability to locate all concepts within each curricular document automatically. We have developed KnowledgeMap (KM) to provide these functions.

BACKGROUND

KnowledgeMap 설명

KnowledgeMap is a web-based application that uses a concept identifier to locate biomedical concepts automatically from search queries of medical education documents.5 Faculty members use the KM web application to upload presentations and lecture handouts in HTML, Microsofts Words, Adobe Acrobat s PDF, and Microsofts PowerPoints formats. As each document is uploaded, the KM concept identifier uses a rigorous algorithm6 to find all Unified Medical Language System (UMLS, a composite vocabulary containing more than one million terms)6,7 concepts located in the document. KM has performed well in identifying biomedical concepts in large sets of medical curriculum documents.6

KnowledgeMap can also search for broad concepts, termed metaconcepts, in the curriculum. Examples of metaconcepts include ‘‘genetics’’ or ‘‘women’s health.’’ After a user submits a metaconcept query, such as ‘‘genetics,’’ the tool constructs a large list of related subconcepts (such as chromosome, point mutation, phenotype, etc.) using relationships defined in the UMLS. A user can decide how many subconcepts he or she wishes to search for by selecting from 6 expansion levels ranging from narrow (fewer subconcepts) to wide (more subconcepts). Once a user has selected the desired number of subconcepts and submits them for a search, KM returns the documents that contain the submitted subconcepts ranked by relevance. Analogous to a web search engine, the output lists all documents matching any subconcepts with the most relevant documents listed first. The user may select a document to see the display of subconcepts located in that document as shown in Figure 1.

METHODS

We tested the ability of KM to locate metaconcepts within the Vanderbilt School of Medicine curriculum. The Institutional Review Board approved the study. First, we determined 4 metaconcepts of interest; next, we established gold standard rankings for a set of documents; and then we searched the document set for metaconcepts. The authors identified 4 metaconcepts relevant for curriculum- type coverage queries: ‘‘genetics,’’ ‘‘women’s health,’’ ‘‘dermatology,’’ and ‘‘radiology.’’ Two of these, ‘‘genetics’’ and ‘‘women’s health,’’ were actual concept queries posed by curriculum review committees at the Vanderbilt School of Medicine.

One author (A.S.) scored 383 documents from 19 firstand second-year medical school courses in order to establish the gold standard set of documents. The author has 12 years’ experience in teaching in all 4 years of medical school and was not familiar with the concept coverage algorithm or the vocabulary sets. He scored each document as 1) having little or no information (‘‘low’’ documents), 2) having a moderate amount of information (‘‘moderate’’ documents), or 3) having a large amount of information (‘‘high’’ documents), which pertains to each of the 4 metaconcepts tested. References and ‘‘suggested readings’’ sections in documents were ignored when ranking. To validate the gold standard rankings, an expert in each field (genetics, women’s health, dermatology, and radiology) scored a set of 10 documents. The percentage of exact agreement between each expert and author A.S. ranged from 90% to 100%; the overall k between the author and 4 experts was 0.84 (calculated with PRAM8).

Using each expansion level of the KM metaconcept search (up to 40,000 subconcepts), we generated relevance scores for each of the 383 documents. We calculated sensitivity and specificity by iterating through all possible score ‘‘cutoffs’’ to identify ‘‘high,’’ ‘‘moderate or high,’’ and ‘‘low’’ documents among all documents scored. Documents with a score of zero were classified as ‘‘low.’’ For the gold standard ‘‘high’’ set, we considered a true positive any document ranked ‘‘high’’ in the gold standard set and above a given score cutoff in KM; ‘‘moderate’’ and ‘‘low’’ documents were gold standard negatives. For the ‘‘moderate-high’’ set, we considered a true positive those documents above a given score cutoff that were ranked ‘‘high’’ or ‘‘moderate’’ in the gold standard set; ‘‘low’’ documents were the only gold standard negatives. We derived a receiver operating characteristic (ROC) curve from the calculated sensitivity and specificity for each expansion set of each metaconcept. To compare the efficacy of each ranking, we calculated the area-under- the-curve for each ROC curve.9 The optimal sensitivity and specificity were determined by finding the northwest corner of each curve. For all statistical calculations, we used Statas version 7 (College Station, TX) and Microsofts Excel (Redmond,WA).

---

J Gen Intern Med. 2005 Oct;20(10):943-6.

"Where do we teach what?" Finding broad concepts in the medical school curriculum.

Denny JC1, Smithers JD, Armstrong B, Spickard A 3rd.

Author information

Abstract

BACKGROUND:

Often, medical educators and students do not know where important concepts are taught and learned in medical school. Manual efforts to identify and track concepts covered across the curriculum are inaccurate and resource intensive.

OBJECTIVE:

To test the ability of a web-based application called KnowledgeMap (KM) to automatically locate where broad biomedical concepts are covered in lecture documents in the Vanderbilt School of Medicine.

METHODS:

In 2003, the authors derived a gold standard set of curriculum documents by ranking 383 lecture documents as high, medium, or low relevance in their coverage of 4 broad biomedical concepts: genetics, women's health, dermatology, and radiology. We compared the gold standard rankings to KM, an automated tool that generates a variable number of subconcepts for each broad concept to calculate a relevance score for each document. Receiver operating characteristic (ROC) curves and area-under-the-curve were derived for each ranking using varying relevance score cutoffs.

RESULTS:

Receiver operating characteristic curve areas were acceptably high for each broad concept (range 0.74 to 0.98). At relevance scores that optimized sensitivity and specificity, 78% to 100% of highly relevant documents were identified. The best results were obtained with the application of 63 to 1437 subconcepts for a given broad concept. The search time was fast.

CONCLUSIONS:

The KM tool capably and automatically locates the detailed coverage of broad concepts across medical school documents in real time. Use of KM or similar tools may prove useful for other medical schools to identify broad concepts in their curricula.