이론 프레임워크 리뷰하기: 이론 적용의 퀄리티 판단과 어려움(Med Teach, 2016)

이론 프레임워크 리뷰하기: 이론 적용의 퀄리티 판단과 어려움(Med Teach, 2016)

Reviews of theoretical frameworks: Challenges and judging the quality of theory application

Sarah Hean, Liz Anderson, Chris Green, Carol John, Richard Pitt & Cath

O’Halloran

BEME리뷰의 목적

Introduction: The purpose of the BEME review

방법론적으로 강건한 리뷰를 수행할 수 있는 가이드를 제공함으로써 보건의료인교육, 의학교육의 이슈에 대한 근거를 수집하고, 평가하고, 통합synthesis할 수 있게 하는 것.

The Best Evidence in Medical Education and Health Professions (BEME) Collaboration supports this decision-making through guiding the conduct of methodologically rigorous reviews that collect, appraise and synthesise the available information and evi- dence associated with a stated issue in health professional and medical education (http://www.bemecollaboration.org/).

인터벤션 이론intervention theory의 개발

Developing an intervention theory in medical and health education practice

메커니즘, 컨텍스트, 성과 이 세가지가 theory of the intervention을 형성한다. 다르게말하자면, theory of the intervention은 논리적 주장argument를 통해 여러 개념을 서로 연결시켜주는 제안의 집합(set of propositions)이다. 이 statement는 특정 현상을 예측/기술/설명/처방/조직화한다.     

The connection between mechanisms, context and outcomes together form the theory of the intervention (Pawson & Tilley 1997). Put another way, the theory of the intervention is a set of propositions that link concepts together through a rational argument. These statements predict, describe, explain, prescribe or organise a particular phenomenon (Fawcett & Downs 1992; Jary & Jary 1995; Walker & Avant 2005).

모든 교육자들은 자신의 intervention theory를 가지고 있다. 이 이론들은 명시적으로든 암묵적으로든 그 intervention이 영향을 미칠 대상이 누구이며, 어떻게 그리고 왜 그 intervention이 그러한 효과를 가져올 것으로 기대되며, 어떤 조건에서 intervention이 특성 성과를 낼 것인지를 예측하게 해준다.

All educators will hold an intervention theory: These theories, whether implicitly or explicitly expressed, predict for whom the intervention will have an influence, how and why the intervention is expected to have this effect and in what conditions the intervention will lead to particular outcomes.

intervention theory를 가지고 있는 것은 어떤 교수-학습에 대해서든 근거기반이 되는 핵심이 된다. 이것은 교육자들이 자신의 교육적 접근을 정의하고/예측하고/조직화하고/설명하고/검증하는데 필수적인 도구이다. 이론과 실천은 상호의존적이며, 이론이 성찰적 교육적행위를 할 수 있는 구조와 강건함rigour을 준다면, 실천은 효과적인 교육은 무엇인가에 대해서 가지고 있는 이론을 검증하고 정제할 수 있게 해준다.

Having an intervention theory, as described above, is centrally important to the evidence base supporting any teaching and learning. It is an essential tool with which educators can define, predict, organise, explain and test their pedagogical approaches. Theory and practice are interdepend- ent, with theory providing the structure and rigour to guide reflective educational practices and in turn, for practice to testout/validate and refine the theories they hold about what constitutes effective education.

IPE를 한 사례로.

A theoretical review of interprofessional education as an illustration

A team of reviewers, members of the In-2-theory (inter- professional theory, scholarship and collaboration) network (Hean et al. 2013) responded to this need for a synthesis of theories in IPE and are in the process of conducting a BEME review (Hean et al. 2012a,b) aiming to provide medical educators guidance on how to select and apply theories to IPE that are fit for purpose and enable educators to reflect on the why, rather than just the how, of designing, delivering and then evaluating the effectiveness of an interprofessional curriculum.

 

이론의 의의: 공통의 이해에 도달하기 위한 수단

The meaning of theory: Reaching a common understanding

'이론'이라는 용어에 대한 공통의 이해가 필요하다. 이 공통의 이해에 도달하는 것은 리뷰프로토콜의 모든 단계에서 평가자간 신뢰도에 핵심이다.

a common understanding of the meaning of the term theory. Reaching this common understanding is key to the inter-rater reliability at all phases of the review protocol.

이론적으로, 공통의 이해에 도달하는 두 가지 전략이 있다. 첫 번째는 boundary crossing이고, 두 번째는 different ways of knowing이다. 모든 리뷰팀은 서로 경험이 다양하고 이론적 역량이 다양한 구성원을 보유하게 된다.

Theoretically, strategies to reach shared understanding rest on two concepts. First, boundary crossing (Carlile 2004) and second, the different ways of knowing articulated by Heron & Reason (2008). Any review team will have members with a range of experiences and varied levels of theoretical compe- tence.

한 개별 이론에 대한 지식과 "이론의 이론"은 리뷰팀 구성원간 서로 transfer되어야 한다.

Knowledge about any one individual theory and the ‘‘theory of theory’’ needs to transfer between members of the review team.

이 boundary를 넘나들기 위해 사용하는 전략은 Heron & Reason’s 의 ways of knowing에 기반하며, 이것은 지식을 습득하는 네 가지 방법에 대해서 다룬 것이다. 위에서 기술한 '이론'의 정의는 propositional(제안) form of knowing이며, 특정 개념에 대한 언어와 기술된 description에 기반한다. 그러나 조만간 이러한 형태의 knowing은 이론을 reliable하게 적용할 때 필요한 공동의 이해에 도달하는데 충분하지 않음이 드러난다.

The strategies used to transcend these boundaries, within the IPE theory review, are rooted in Heron & Reason’s (2008) ways of knowing where four ways of acquiring knowledge are represented. The definition of theory written above is a propositional form of knowing, that is based on language and written description of the concept. However, it was soon evident that this form of knowing was not enough to reach a common understanding of theory that could be reliably applied.

이에 우리는 또 다른 ways of knowing을 사용했는데, 이는 experiential and practical ways of knowing이다. 리뷰팀 구성원들은 과제를 함께 수행하며 합의에 도달했는데(practical knowing) 예컨대 다음의 것들을 함.

We turned therefore to other ways of knowing for assistance: namely experiential and practical ways of knowing. Review members reached a consensus by doing tasks together (practical knowing); by holding regular workshops using exemplar pilot papers to work through together as a team during paper selection, quality assessment and extraction phases of the review.

 

경험학습Experiential learning은 특정 이론에 더 경험이 많은 리뷰팀 구성원들이 경험이 적은 구성원과 짝을 이뤄서 일을 하면서 특정 문헌에 대해서 설명articulate해주고, 그럼으로써 경험이 적은 구성원이 experientially 배울 수 있는 것이다. 이 때 시각화전략을 사용하여서 경험이 많은 reviewer가 그 문헌에서 theory로서 발견한 것을 highlight하여 경험이 적은 파트너는 어떻게 자기의 파트너가 이론을 받아들이고conceiving있는지 알 수 있었다.

Experiential learning also took place when review members, more experienced in theory or a particular theory, worked together in pairs with less experienced members, articulating the review of any one particular paper, so that the less experienced member was able to learn experientially. Visualisation strategies were used at these times whereby the experienced reviewer highlighted what they had identified as theory in this article, so less experienced partners could see how their partner was conceiving theory, rather than relying on their verbal description thereof (Figure 1).

 

튼튼한 공동의 이해를 위한 또 다른 전략은 practical and experiential knowledge를 propositional knowledge로 다시 translate back하는 것이다. 이 때 팀원들은 (이론에 대한 자신의 이전 지식에 experiential and practical한 이해를 합해서) 이론의 의미가 무엇이 되어야 한다고 생각하는지를 성찰하여 써 보았다(propositional knowledge).

Another strategy to cement shared understanding of theory involved the translation of practical and experiential know- ledge back into propositional knowledge. Hereby, members of the teamwere asked to write their reflections on what they felt the meaning of theory to be (propositional knowledge) drawing on their own previous knowledge of theory but also their understanding developed in their working experientially and practically with other members of the team in pairs and in workshops (Figure 2 for an example of one reviewer’s reflection).

지식을 미적aesthetic인, 표현적인 방법(춤 등)으로 공유하는 것은 Heron and Reasons 이 제시한 마지막 ways of knowing으로 presentational knowing이다. '이론'에 대해서 이것은 아마 어려운 요구이겠지만, 공동의 이해에 도달하는데 있어서 rule out 되어서는 안 될 것이다.

Sharing knowledge through aesthetic, expressive methods such as art or dance, the final of Heron and Reasons (2008) ways of knowing (presentational knowing) is perhaps a tall order for a concept such as theory but should not be ruled out as means of reaching common understanding between participants.

 

 

 

이론의 적용의 퀄리티 평가

Assessing the quality of application of theory

 

특정 교육적 intervention을 지지하는 근거에 대한 전통적인 리뷰에서, 방법론적인 퀄리티를 평가하는 것은 sufficient rigour를 갖춘 문헌을 선택하는 핵심이었다. review of theory는 이러한 측면에서 조금 다르다. 이 때 초점은 방법론적 퀄리티가 아니라 특정 커리큘럼이나 커리큘럼의 평가를 뒷받침하는데 사용된 이론의 퀄리티를 보는 것이다.

In a conventional review of the evidence supporting a particular educational intervention, the assessment of methodological quality is key to the review team selecting papers of sufficient rigour so as to constitute good evidence in the field. A review of theory is distinct in this regard as the review is essentially exploring the effective use of theory in educational interventions. The focus is no longer on methodological quality but the quality with which theory has been used to underpin the curriculum or its evaluation 

이론 그 자체의 퀄리티가 아니라, 그것이 적용된 방식의 퀄리티이다.

It is important to note here that it is not the quality of the theory itself that is in question but the quality of its application.

방법론적 퀄리티를 평가하는 프레임워크는 흔하지만, 이론적 퀄리티에 대한 척도는 드물다.

While frameworks for assessing methodological quality are common place (BEME Collaboration 2012; CASP 2012) measures of theoretical quality are absent in systematic review methods.

이러한 이유로 이론적 퀄리티 측정을 위한 프레임워크를 만들었다.

For this reason, the IPE review team developed own framework to measure theoretical quality. The dimensions of this framework originate from those developed by Fawcett and Downes when assessing the links between theory and research (Fawcett & Downs 1992; Fawcett 2005), namely,

• the dimensions of parsimony,

• prag- matic adequacy,

• testability,

• operational and

• empirical ade- quacy.

 

The meanings of each of these are discussed below.

 

실용적 적합성 (실천에 대한 이론의 유용성이 명확한가?)

Pragmatic adequacy (is the utility of the theory for practice made clear?)

practitioner를 위한 확실한 유용성이 있어야 한다. 논문에서는 그 이론이 누구에게 유용하며, 어떻게 그 이론을 working practice에 도움이 되게 할 수 있는지를 설명해야 한다.

Pragmatic adequacy means that the theory has clear utility for practitioners: when reading the report of the curriculum or its evaluation, practitioners should be able to see clearly how the theory can inform their working lives. By practice, we mean the theory must have been used to underpin an interprofessional curriculum, the way it is delivered and/or the approach taken to its evaluation (Coles & Grant 1985). Papers need to explicitly address the questions of who is the theory useful for and how can they use this theory to inform their working practice.

실용적 적합성은 논문을 고르는 중요한 기준 중 하나이다. 매우 흥미로우면서 학문적으로도 이론을 탐구하거나 개발하는 sophisticated한 논문들이 많으나, 만약 고찰이 추상적인 수준에서 머물러서 교육/연구 practice에 대한 link가 없으면 더 이상 review되지 않는다.

Pragmatic adequacy is a key selection criterion for papers. There are many highly interesting and academically sophisticated papers exploring and developing theory but if the discussion remains at an abstract level with no clearly articulated link to education or research practice, then the paper should not be reviewed further.

 

절약(이론은 간결하고 명확히 기술되었는가?)

Parsimony (is the theory concisely and clearly described?)

이론이 잘 기술되지 않는 경우가 많다. 이론적 기반이 없거나 명확하게 드러나지 않았거나.

Parsimony means that theories are clearly but concisely described in reporting. In many reports on IPE curricula, for example, the theory underpinning the curriculum is not described. It is uncertain then if the curriculum has no theoretical foundation , or whether this theory was simply not made explicit.

그러나 과도한 단순화를 경계해야 한다.

Authors on the other hand must guard against an over simplification of the phenomenon being addressed (Fawcett & Downs 1992; Fawcett 2005).

 

시험가능성(이론으로부터 유도된 명확한 제안propositions이 명확하게 제시되었는가?)

Testability (are clear propositions, derived from the theory, clearly presented?)

이론의 각 요소들을 형성하는 주요 개념과, 그 개념들 사이에 대해 제안되는 관계가 명확히 기술되어야 한다.

Key concepts that form the components of the theory and the proposed relationships between them should be clearly articulated.

• 예시: Therefore, for example in Table 1, the three main principles of complexity theory are spelt out (i.e. that in these systems, knowledge is emergent, that the system is self- organising, more than the sum of its parts and nested) in addition to the five conditions required for collective learning to take place within this complex system (e.g. decentralised control, internal diversity).

운영상의 적합성(그 제안propositions들과 test에 사용된 연구방법 혹은 이론을 사용하는 방식 사이에 명확한 link가 있는가?)

Operational adequacy (is there a clear link between the propositions and the research method used to test or use the theory?)

이 criterion은 어떤 제안/가설이 이론으로부터 만들어지면, 검증되고 적절한 연구디자인과 방법으로 apply된다.

This criterion is fulfilled if the proposition/hypothesis created from the theory is then tested or applied with an appropriate research design and method.

• 예시: Weaver et al. (2011) (Table 1) for example, clearly demonstrate how the dimensions of com- plexity theory and learning conditions have developed an analytical framework for a content analysis of the transcripts.

경험적 적합성

Empirical adequacy

 

실제 수집한 empirical data가 그 연구를 지탱하는 이론과 부합하는지

This means that the empirical data collected is congruent with the theory that underpins the study.

 

현실에서는 위의 dimension들은 서로 매우 강력한 상호의존성을 보이고 있다. 예컨대 pragmatic adequacy가 높은 경우 나머지도 좋다.

In reality, the above dimensions of theoretical quality described here are strongly interdependent, i.e. papers that have the most pragmatic adequacy are those that

• are clearly and simply articulated (parsimony),

• lay out clear propositions (testability),

• use appropriate methods to derive or test these propositions (operational adequacy)

• then report data that is congruent with the theoretical approach being utilised (empir- ical adequacy).

Conclusion

We recommend that the theory quality assessment tool (Table 1) be applied to future theory reviews. It should also be included as an extra quality dimension in the protocols of rigorous literature reviews more widely.

BEME reviews are practi- tioner-focussed endeavours, and hence the focus on pragmatic value as a key inclusion criterion. We acknowledge this means the loss of many papers that still have high academic value and that make a contribution to theoretical development in the field. Pragmatic value is the only exclusion criterion and tool deliberately does not excluded non-empirical papers. Although these papers will score low on operational and empirical adequacy, an articulation of the intervention theory and howthese apply to education practice, still have pragmatic value for the reader.

 

 

---

Med Teach. 2016 Jun;38(6):613-20. doi: 10.3109/0142159X.2015.1075648. Epub 2015 Oct 16.

Reviews of theoretical frameworks: Challenges and judging the quality of theory application.

Hean S1,2, Anderson L3, Green C4, John C5, Pitt R6, O'Halloran C7.

Author information

• 1a Bournemouth University , UK .
• 2b University of Stavanger , Norway .
• 3c University of Leicester , UK .
• 4d University of Essex , UK .
• 5e The Anglo-European Chiropractic College , UK .
• 6f University of Nottingham , UK .
• 7g University of Huddersfield , UK.

Abstract

BACKGROUND:

Rigorous reviews of available information, from a range of resources, are required to support medical and health educators in their decision making.

AIM:

The aim of this article is to highlight the importance of a review of theoretical frameworks specifically as a supplement to reviewsthat focus on a synthesis of the empirical evidence alone. Establishing a shared understanding of theory as a concept is highlighted as a challenge and some practical strategies to achieving this are presented. This article also introduces the concept of theoreticalquality, arguing that a critique of how theory is applied should complement the methodological appraisal of the literature in a review.

METHOD:

We illustrate the challenge of establishing a shared meaning of theory through reference to experiences of an on-going review of this kind conducted in the field of interprofessional education (IPE) and use a high scoring paper selected in this review to illustrate how theoretical quality can be assessed.

FINDINGS:

In reaching a shared understanding of theory as a concept, practical strategies that promote experiential and practical ways of knowing are required in addition to more propositional ways of sharing knowledge. Concepts of parsimony, testability, operational adequacy and empirical adequacy are explored as concepts that establish theoretical quality.

CONCLUSIONS:

Reviews of theoretical frameworks used in medical education are required to inform educational practice. Review teams should make time and effort to reach a shared understanding of the term theory. Theory reviews, and reviews more widely, should add an assessment of theory application to the protocol of their review method.

PMID:

 

26473273

 

DOI:

 

10.3109/0142159X.2015.1075648

[PubMed - in process]