임상과학자프로그램 평가 - 커크패트릭을 넘어서. "효과가 있었나"와 "어떻게 효과가 생겼나" (Acad Med, 2011)

임상과학자프로그램 평가 - 커크패트릭을 넘어서. "효과가 있었나"와 "어떻게 효과가 생겼나" (Acad Med, 2011)

Going Beyond Kirkpatrick in Evaluating a Clinician Scientist Program: It’s Not “If It Works” but “How It Works”

Kathryn Parker, PhD, Gwen Burrows, MA, Heather Nash, and Norman D. Rosenblum, MD

의학교육계에서 (프로그램평가는) 흔히 프로세스보다는 Outcome을 강조한다.

In medical education, commonly used models and approaches emphasize the outcomes rather than the processes that lead to outcomes.1

여러 근거를 보면, well-constructed한 프로그램이라도 고등 수준의 행동변화를 얻어내는데 실패하는 경우가 많으며 이는 심지어 sophisticated learner에 대해서도 마찬가지다.

Evidence suggests that programs—including well-constructed ones—fail to demonstrate outcomes at the higher levels of behavioral change, even in sophisticated learners.7–11

Outcome-focused 모델은 프로그램의 이해관계자들에게 프로그램의 rationale, goal, 의도한 목표에 대한 정보 뿐만 아니라 그러한 성과를 내기 위한 input와 process에 대한 정보도 제공한다. 그러나 이러한 모델은 "프로그램 이론"에 대한 통찰은 거의 주지 못한다. 또한 의도하지 않은 성과를 잡아내지도 못한다. 따라서 성과-지향적 모델은 high-level의 프로그램 성과를 가능하게 했거나 불가능하게 만들어준 기저에 깔린 메커니즘을 이해하는데 거의 통찰을 주지 못한다.

Outcomes-focused models may provide program stakeholders with a common knowledge of the program’s rationale, goal, and intended outcomes—as well as the inputs and processes required to arrive at those outcomes. Yet, these models provide little insight into a program’s “theory”—that is, the processes by which it yields desired outputs and outcomes—and they do not capture unintended outcomes.12–14 Accordingly, outcome-focused models provide little insight into the underlying mechanisms that hinder or enable the achievement of higher-level program outcomes.

우리는 emergent and unintended outcome을 평가하는 것이 어떻게 교육프로그램을 usual, outcome-based method를 넘어서는 성과를 주는지를 평가해보고자 했다.

We investigated how evaluating emergent and unintended outcomes can informan educational program in ways that extend beyond the results of usual, outcome- based methods.

프로그램

The Program

CCHCSP : 의사과학자 양성 프로그램

The overarching objective of the CCHCSP is to educate a new generation of clinician scientists in child and youth health within an interdisciplinary model.

17개의 AHC

The CCHCSP is a partnership among 17 Canadian child-and-youth-health- focused academic health centers (AHCs).

주요 커리큘럼

CCHCSP trainees engage in a curriculum which consists of four major components:

• Web- based learning modules,

• interdisciplinary mini-symposia,

• an annual national symposium, and

• international meetings.

 

온라인 학습모듈(5개)

The five online learning modules focus on the following subject areas or content:

• (1) research ethics and conflicts of interest,

• (2) research design and clinical research practice,

• (3) managing one’s career as a clinician scientist,

• (4) oral and written communication, and

• (5) knowledge translation.

미니심포지엄

The CCHCSP uses research exercises at mini-symposia as key elements for learning. A typical symposium focuses on a specific theme (e.g., “collaborating in research groups”) and consists of a series of keynote lectures, working seminars that address the theme, and small-group sessions that focus on pivotal questions, which again relate to the theme.

연례 심포지움

The annual national symposium, which takes place in a different Canadian city each year, also features workshops, but these focus on specific skill sets relevant to the emerging clinician scientist.

방법

Method

로직-모델 기반 평가

Logic-model-based evaluation of the CCHCSP

Soon after its creation, the CCHCSP developed an evaluation framework using a program logic model. This logic model (Chart 1) was created to serve as a communication tool for the program, to inform program development, and to serve the evaluation needs of program developers and funders.

 

 

 

정량적 평가

Quantitative assessment

Tools developed through the program’s logic model track whether graduates were sufficiently exposed to and engaged in the programto learn and develop transdisciplinary research skills. We examined participation rates in all program components.

정성적 평가

Qualitative assessment

평가위원회의 위원장

To further explore the achievement of goals, one of us, the evaluation committee chair of the CCHCSP (G.B.), conducted confidential, one-on-one, face-to-face exit interviews, each lasting about 30 minutes, with all program graduates, as well as with trainees who did not complete the program(see Appendix 1 for the exit interview questions). The interviewer (G.B.) summarized the resulting data using grounded theory methodology,16 and then we grouped and analyzed recurring themes.

결과

Results

샘플, 참여, 만족도

The sample, participation, and satisfaction

From 2003 to 2009, the program graduated 21 trainees, each of whom received training in one of seven different clinical disciplines: medicine (11 graduates), nursing (2), physical therapy (1), psychology (1), occupational therapy (2), speech language pathology (3), and dentistry (1).

다수가 네 개의 프로그램 요소에 참여

A large number of graduates engaged in each of the four program components (Table 1).

만족도는 높음.

Graduate satisfaction with the mini- symposia, as well as with the seven national symposia held between 2003 and 2009, was high. The average overall rating by all participants (including graduates) was 4.33 out of 5; the range was from a low of 4.2 in 2004 to a high of 4.5 in 2006.

 

 

 

프로그램 성과

Program outputs and outcomes

생산성

Productivity.

연구논문 출판, 초청발표, 펀딩

The number of peer-reviewed publications they produced, the number of invited presentations they gave, and the amount of funding they garnered (Table 2)

 

학제간 연구와 커리어

Interdisciplinary research and careers.

학제간 연구에 참여하였음.

Graduates’ research projects indicated that they engaged in interdisciplinary research as established investigators (a program outcome articulated in the logic model); the mean number of disciplines their research represented was 3.2, and the rangewas 2 to 8.

 

 

질적 탐구

Qualitative inquiry

 

학제간 연구에 대해 노출된 것이 중요한 프로그램 성과 중 하나였다. 더 나아가서 졸업생들은 CCHCSP가 연구 프로세스와 시스템에 대한 시야를 넓혀주었으며, 전문직 네트워크과 서포트 시스템을 늘려주었다고 하였음.

Table 3 provides a summary of the themes that emerged from the interviews regarding progra mgraduates’ experiences with and perceptions of the CCHCSP. We found that the ability of the CCHCSP to expose graduates to interdisciplinary research was a critical program outcome. Furthermore, graduates indicated that the CCHCSP broadened their perspectives about research processes and systems and increased their professional networks and support systems.

 

 

 

 

의도하지 않은 성과와 추가 탐구

Unintended program outcome and further inquiry

21명의 졸업생 중 14명이 프로그램이 자신들의 커리어를 준비할 수 있게 도와주는데 도움이 되었던 것으로 의사과학자의 역할을 이해하고 명확하게 해준 것을 꼽았다. 이는 의사과학자로서 자신을 이해하는데 기여하였다. 일부 졸업생들은 이러한 성과를 전문직 정체성 변화로 연결시켰다Professional identity alteration.

Fourteen of the 21 graduates (67%) said that one of the most important ways the program prepared them for their careers was to provide understanding and clarity about the clinician scientist role, which in turn contributed to their understanding of themselves as clinician scientists. Some graduates linked this outcome explicitly to professional identity alteration (Table 3, bottom row).

그러나, 면담에서의 정보는 값어치있는 것이긴 했지만 어떻게 프로그램이 의사과학자로서 self-identification을 향상시켰는지에 대해선 충분히 설명해주지 못한다. 

However, information from the interviews, although invaluable, was insufficient to explain how the program worked to effect enhanced self- identification as a clinician scientist.

앞서 언급한 것처럼, 우리는 문헌조사를 함.

As mentioned above, we conducted a search of the literature to find further information about professional identity.

그러나 우리는 기존 문헌에서 정립된 정체성이 있는 전문직이 새로운 정체성을 얻는지에 대한 이론을 찾지 못했다. 

However, we could not find any theory in the published health care literature on how professionals with an established identity acquire a new one—that is, how practicing clinicians may come to see themselves as clinician scientists.

Ibarra의 임시적-자기 이론

Ibarra’s theory on provisional selves

Herminia Ibarra의 임시적-자기 이론은 전문직은 "(아직 완전히 형성되지 않은 전문적 정체성인)임시적 자기"를 가지고 pseudo-trial- and-error process (유사 시도-실패 과정)을 거치며 새로운 정체성을 얻게 된다. 임시적 자기는 전문직이 새로운 역할을 가지게 되면서 constructed, rehearsed, and refined 되는 전문직 정체성이다. 임시적-자기는 다음의 시기에서 '임시적-자기'가 형성constructed 된다고 말했다.

Herminia Ibarra’s28 theory on provisional selves proposes that professionals adopt new identities through a pseudo-trial- and-error process of experimenting with “provisional selves” that are possible, yet not fully formed, professional identities. Provisional selves are new professional identities that are constructed, rehearsed, and refined as a professional takes on a new role. This provisional self is constructed during

“times of career change or transition, as people identify role models, experiment with unfamiliar behaviors, and evaluate their progress.”28

 

Ibarra는 전문직은 새로운 역할을 받아들이는 경우 세 개의 단순한 task에 참여하게 된다고 주장했다.

Ibarra asserts that professionals will engage in three simple tasks in the adaptation of new roles:

• (1) 새로운 정체성을 찾기 위한 롤모델 관찰 observing role models to identify possible identities,

• (2) 임시적-자기를 가지고 실험함 experimenting with provisional selves, and

• (3) 이러한 실험을 내적 기준과 외적 피드백에 대해 평가함 evaluating these experiments against internal standards and external feedback.

CCHCSP를 위한 새로운 프로그램 이론.

A new program theory for CCHCSP

우리는 로직모델 데이터를 통합하고 문헌 결과를 합하여 전문직-정체성-변화에 대한 새로운 프로그램 이론을 도출하였다.

We merged logic model data (including qualitative information from exit interviews) and findings from the literature on professional identity change to articulate a new program theory or model for the CCHCSP (Figure 1).

 

우선 세 가지 critical components 를 통합하여 trainee는 의사과학자로서 자신을 찾아간다.

We hypothesize that the process by which this program works to enable a trainee to identify as a clinician scientist is first a synthesis of three critical components (Figure 1, Boxes 1–3).

• 커리큘럼에 engage함. 다양한 전공과 연구분야의 동료들을 포함함.
  First, the trainee needs to engage in the curriculum(Figure 1, Box 1), which includes interacting with a group of peers from diverse clinical and research disciplines.

• 이는 연구 성과로 이어짐
  Engagement in the curriculum, in turn, contributes to an increased productivity in research (Figure 1, Box 2).

• 미니 심포지엄, 연례 심포지엄, 국제 미팅 등에서 정기적으로 결과물을 발표하고, 연구 프로세스에 참여함으로써 롤모델(멘토)를 관찰하고 함께 일할 기회를 얻게 됨. 이는 Ibarra의 이론에 따르면 임시적-자기로 실험을 하는 것과 같다.
  Engagement in the mini- symposia, the annual national symposium, and the international meetings where trainees regularly present their research, as well as engagement in the research process, provides the opportunity for trainees to work with and observe role models (mentors) which, according to Ibarra’s theory, enable the trainee to experiment with provisional selves (Figure 1, Box 3).

그 다음, trainee들이 이 '실험'을 할 때 (이 모델에서는 명확하게 밝혀지지 않은) 세 가지 source가 각 임시적-자기를 평가하는데 필요한 정보를 준다.

Next, as the trainee engages in this experimentation, three sources (not identified explicitly in the model) provide information to the trainee that informs the evaluation of each provisional self.

• 외부 피드백: 다양한 교수와 연구자들로부터의 피드백
  The trainee evaluates his or her experiences against external feedback provided by various faculty or researchers. This external feedback (formal or informal in nature) may come from the trainee’s mentor, a peer, another faculty member, or from the program leaders.

• 자기 스스로의 기준(내적 기준)
  The trainee also evaluates each provisional self against his or her own internal standards. If the provisional self is in accordance with these internal standards, full or partial adoption of the provisional self is more likely to occur.

• 외적 기준. 프로그램이 정해준 기준, 시스템이 정한 기준, 여러 활동(투고, 연구비 수주 등)에서 얻은 기준. 이 시도-실패 과정은 반복적이고 역동적이다. 이 과정을 거치며 내적 기준이 변화하거나 강화된다.
  Finally, the trainee evaluates each provisional self against external standards. External standards may be those set by the program—but they may also be those that are set by the system in which a clinician scientist works and/or those that are learned through activities such as applying for grants, submitting manuscripts for publication, and seeking jobs. This trial-and-error process is an iterative and dynamic one. As each provisional self is evaluated against external feedback and internal standards, the trainee’s internal standards may be altered or reinforced. 

 

4. 이 평가 프로세스는 trainee의 연구에 대한 자신감과 흥미에 직접적으로 영향을 준다.

This process of evaluation directly influences the trainee’s confidence and interest in pursuing research (Figure 1, Box 4) and

5. 그 다음 의사과학자의 역할과 시스템의 역할에 대해 명확하게 이해하게 된다.

generates clarity of the role of the clinician scientist and of the system in which a clinician scientist works (Figure 1, Box 5).

 

6. 이 자신감/흥미는 trainee가 의사과학자가 될 것인지에 대한 결정에 중요하다.

This confidence/interest and this clarity are components necessary for the trainee to determine if he or she will identify as a clinician scientist (Figure 1, Box 6). (Notably, these two constructs— confidence/interest and role clarity— were also identified by the program’s logic model and exit interview data.)

7. 적극적으로 employment를 찾고

The trainee who identifies as a clinician scientist will actively seek employment in the role (Figure 1, Box 7), thus

 

8. 차세대 의사과학자로서 기여한다.

contributing to the next generation of clinician scientists (Figure 1, Box 8).

 

 

Table 3 Experiences, by Theme, of the 21 Program Graduates of the Canadian Child Health Clinician Scientist Program as Identified in Exit Interviews, 2003–2009

고찰과 결론

Discussion and Conclusions

인터뷰에서 수집된 정보는 Ibarra의 임시적-자기 이론과 부합한다. 두가지에서 공통적이었던 것은 전문직으로서의 정체성을 형성하거나 변화시키는데 중요한 것은 "네트워크와 멘토"라는 점이다. Ibarra는 "멘토"는 그 자체로 possible provisional self라고 했다. 즉, trainee가 멘토의 다양한 특징을 받아들이기 전에 시도(tries on)해 본다는 것이다. 비슷하게, 인터뷰에서도 멘토링과 네트워킹이 의사과학자로서 무엇이 필요한가를 이해하는데 핵심이었다. 비록 프로그램이 명시적으로 강력한 멘토링 요소를 포함시키기는 했지만, 보다 비공식적으로 이루어진 멘토링의 중요성이 확인되었다. Ibarra의 주장과 우리의 자료에서는 이 멘토-멘티 관계가 commonly held understandings를 전달하는데 중요하며, 의사과학자 커뮤니티의 문화적 규범은 졸업생이 자신을 의사과학자로서 정체성을 갖게 되는 과정에서 중요하다. 이러한 "잠재 교육과정"을 이해하는 것은 어떻게 이 프로그램이 "관측된 성과"를 가져오는지 이해하는데 중요하다.

Information that we collected from the interview data is consistent with assumptions in Ibarra’s theory on professional identity. Common to both is the importance of the network and mentor in establishing or changing a professional identity. Ibarra asserts that the mentor is a possible provisional self: that the trainee “tries on” various characteristics of the mentor prior to the adoption of any of these characteristics. Similarly, exit interview data indicate that the mentoring and networking provided by the programare key to achieving a greater understanding of what is required as a clinician scientist. Although the programexplicitly built in a strong mentoring component, the importance of more informal mentoring was also clearly evident. Ibarra asserts, and these data support, that this mentor–mentee relationship is critical for conveying commonly held understandings about, and the cultural norms of, the clinician scientist community, which are in turn vital to the process of graduates identifying themselves as clinician scientists, members of the community. Understanding this “hidden curriculum” (implicit in many programcomponents) provides insight into how this program works to bring about the observed outcomes.

우리의 결과와 Ibarra의 이론의 또 다른 공통점은 role clarity의 중요성이다. Ibarra는 trainee들은 임시적-자기를 시도해봄으로써 그 직업의 현실을 보다 잘 이해하고, 그 안에서 자신의 위치를 이해하게 된다. Role clarity는 면접에서 두드러지게 나타났다. 이 clarity를 얻는 과정을 거치며 trainee는 의사과학자가 되는 것이 바람직하지 않은 것이라고 판단내렸을 수도 있다. 이 성과는 프로그램의 로직 모델에서는 거의 다뤄지지 않지만, 프로그램의 실패를 errorneously view하는 것일 수 있다. 교육자들은 종종 그들의 교육적 노력이 의도한 성과를 내지 않으면 실패했다고 말한다. 사실 학습자가 성장과 발달의 과정에서 교육받기 이전에는 깨닫지 못했던 것을 깨닫게 해주는 능력(학습자는 자신이 환자를 보는 것을 더 좋아한다는 것을 깨달음)도 프로그램 성공의 지표가 될 수 있다.

Another commonality between our results and Ibarra’s theory is the importance of role clarity. Ibarra asserts that trainees try on provisional selves to gain a better understanding of the reality of the profession and their place in it. Role clarity was a dominant theme in the exit interviews. Through this process of obtaining clarity, a trainee might come to the conclusion that being a clinician scientist is not a role that is desirable. This outcome, rarely articulated in a program’s logic model, might be erroneously viewed as an indicator of the program’s failure. Educators often view their educational efforts as failures if predetermined outcomes (e.g., a participant does not pursue work as a clinician scientist) are not achieved. In fact, their ability to influence learners in their process of growth and improvement in ways that are not known a priori (e.g., a learner comes to realize she prefers to spend the majority of her time with direct patient care) should be viewed as an indicator of program success.

프로그램 개발자와 평가자로서 우리는 세 가지 핵심 교훈을 얻었다.

As program developers and evaluators, we learned three key lessons through this process.

• 로직모델을 개발하는 것은 Sound 프로그램 기획과 평가의 부분이다.
  First, the development of a program’s logic model is part of sound program planning and evaluation. A small but worthwhile investment, it brings knowledge of the program to its key stakeholders and should be created at the outset.

• 프로그램이론을 설명하는 것은 프로그램의 의도한 성과와 의도하지 않은 성과에 대한 이해를 준다
  Second, the articulation of a program’s theory provides a greater understanding of how the program brings about not only predetermined and intended outcomes but also unintended or emergent outcomes. Articulating the underlying theory is vital to program improvement and knowledge generation.

• 프로그램 개발자의 open-ended level of inquiry와 a readiness to integrate the right people and perspectives 가 중요하다.
  Finally, the ability of program developers to bring both an open-ended level of inquiry and a readiness to integrate the right people and perspectives (e.g., theories, including those from the literature of other fields) leads to new hypotheses and research in educational programming.

 

Appendix 1

Exit Interview Questions Asked of Participants, Including Graduates, of the Canadian Child Health Clinician Scientist Program (CCHCSP), 2003–2009

Preamble

The purpose of this exit interview is to get a sense of your experience as a trainee within the CCHCSP, and your views on the strengths and weaknesses of program. Your answers will be kept confidential unless you give me permission to share specific feedback with [other investigators].

Over time I will be aggregating the information from the interviews and seeing if certain themes emerge.

1. What is your experience of the program? Please consider the following:

a. Your specific center and the activities at that center

b. The curriculum (Was it useful? What are some ways it can be improved?)

c. The interdisciplinary experience (Was that valuable? In what ways can it be improved?)

d. The national and mini symposia (Were they valuable? In what ways could they be improved?)

e. The mentorship you received (How was it different from other mentorship you have experienced? Were there any specific qualities to that relationship because it was part of the program?)

f. Salary (Was it important factor in your applying to CCHCSP? Why/why not?)

2. Are there any issues which you think the program needs to address and resolve? Are there any major problems with the program from your perspective?

3. What are the greatest strengths of the CCHCSP from your perspective?

4. How do you think having trained through CCHCSP has prepared you for your career as a clinician scientist?

5. [When relevant] Why did you choose to leave the program before completing it?

---

Acad Med. 2011 Nov;86(11):1389-96. doi: 10.1097/ACM.0b013e31823053f3.

Going beyond Kirkpatrick in evaluating a clinician scientist program: it's not "if it works" but "how it works".

Parker K1, Burrows G, Nash H, Rosenblum ND.

Author information

• 1Holland Bloorview Kids Rehabilitation Hospital, and Department of Paediatrics, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada. kparker@hollandbloorview.ca

Abstract

PURPOSE:

To explore how the Canadian Child Health Clinician Scientist Program (CCHCSP) works to achieve prearticulated and emergent outcomes.

METHOD:

In 2009, after gaining ethical approval from the Hospital for Sick Children, the authors examined quantitative data (e.g., participation in curriculum elements) to ensure sufficient exposure by trainees to the program and quantitative outputs (e.g., publications) to measure achievement of CCHCSP goals. They identified emergent outcomes through grouping and analyzing qualitative data generated through interviews with program graduates. Then, to explore possible theoretical explanations for the emergent findings, the authors conducted a literature review.

RESULTS:

Graduates participated in high rates in each component of the CCHCSP and produced publications, presented research, and received funding. Interview data revealed an unexpected outcome: that the CCHCSP helped graduates to form new professional identities. These data, along with theoretical assumptions from Ibarra's theory on professional identity change, informed a new theory or model for the CCHCSP.

CONCLUSIONS:

Early investment in building a program's logic model is invaluable for understanding program goals and for guiding program planning and development. Both employing a strategy that captures emergent program outcomes and investigating (e.g., through a literature search) why and how the program actually works to arrive at these outcomes informs the development and evaluation of future program offerings and may, as in the case of the CCHCSP, offer a new program model or theory.

PMID:

 

21952054

 

DOI:

 

10.1097/ACM.0b013e31823053f3

[PubMed - indexed for MEDLINE]