Understanding, using and calculating effect size

 Effect size … allows us to move beyond the simplistic, “Does it work or not?” to the far more sophisticated, How well does it work in a range of contexts?” (Coe, 2002)

 

(http://lovestats.wordpress.com/dman/)



What is effect size?

l  a simple measure for quantifying the difference between two groups or the same group over time, on a common scale. (두 그룹간 차이에 대한 척도)

l  Educational setting

n  Effect size is one way to measure the effectiveness of a particular intervention (특정 개입의 효과)

n  By taking into account both improvement and variation it provides information about which interventions are worth having. (어떤 개입방법(intervention)이 가치가 있는지에 대한 정보제공)

 

How is effect size calculated?


 

How can we use effect size?

l  Investigate the effectiveness of a particular intervention for a defined group of students
(특정 그룹에 대한 특정 개입이 갖는 효과를 조사)

l  Compare the effectiveness of different interventions (서로 다른 개입방법 사이의 효과를 비교)

l  Evaluate the growth over time (시간에 따른 성장을 평가)

 

Example

(1) Dr John Hattie said, “for students moving from one year to the next, the average effect size across all students is 0.40.”

(2) 2010 5학년 – 2011 6학년 비교 : effect size 0.49 ( A, B, C 0.86, 0.42, 0.18)

(3) Question :

l  How well is what I am doing working for different groups of students each year and why?

l  What possible reasons could there be for some student or groups of students progressing more or less?

l  How does student progress compare with their achievement levels

 

How can effect size be used reliably?

Multiple measures are still required

l  Effect size is only a single measure of progress (발전과정을 측정하는 하나의 도구일 뿐)

l  Educators should use a range of learner achievement and multiple measures of data (다양한 측정치를 활용해야)
(demographic, perception, student learning and process measures about the teaching and learning environment)


Caution for all small sample sizes and at the individual student level

l  Effect size for cohorts smaller than 30 are often not suitable (n수가 30보다 적은 경우에 효과크기는 적절하지 않을 수도 있다)

l  Individual level effects must always be used in addition with other reliable information and teacher professional judgement (개개인에 대한 효과는 반드시 다른 신뢰할 수 있는 정보, 교수자의 판단과 함께 활용되어야 함)


Accuracy is enhanced when comparing the exact same group of students

l  When comparing pre-test and post-test scores, it is most useful to ensure that all students are tested and that scores from exactly the same group of students are compared (사전 사후 시험점수를 비교하는 경우 정확히 동일한 그룹의 학생이 비교되어야 함.)


NAPLAN effect sizes cannot be compared equally

※ The National Assessment Program – Literacy and Numeracy (NAPLAN) is an annual assessment for students in Years 3, 5, 7 and 9. It has been an everyday part of the school calendar since 2008.

l  There are larger effect sizes for Year 3-5 than in Year 5-7 and Year 7-9.

l  Students at lower proficiency bands - Students in higher proficiency bands

 

In summary, it is important to base the interpretation of effect size on the full range of contextual and measurement factors. This measure is best used to raise questions in conversation and stimulate discussion, particularly around the possible reasons for differences and the question:

What positive difference are we making for this group of students?”

 

+ Recent posts