Research: The Educational BS Repellent

I don’t like BS.  I don’t like it when people make bold, loud claims about different aspects of education, about what works and what doesn’t.  What is even more troublesome for me than these controversial statements is when I don’t have the research backing to quantitatively and qualitatively refute the BS.  After many negative experiences challenging people on their beliefs without backing of my own, I have determined that people cling strongly to these thoughts (whether they are BS or not) unless you can provide research to them that might change their mind.  And really, if you are questioning a set of beliefs with nothing more than your own beliefs, then you are a BSer just as much as the next person.

As a result, over the past several years, I have become a research junkie. I love reading studies about pretty much anything educational.  Homework–does it make a difference?  How about class size?  Failing a student?   But the one thing that I find with reading research is that it takes a great deal of time, and it seems that for each study you find that supports one side of an argument, there are two or three that support the other side of the argument: often, you end up walking away wondering which study to believe. 

However, I have found a new educational Bible for me.  It is called “Visible Learning”, by John Hattie, and it is a compilation of more than 800 meta-analyses of different factors that influence student achievement.  And remembering that meta-analyses are the studies of all of the studies, they tend to remove some of the bias that may come from individual studies. And as far as I am concerned, this book represents what I am going to call educational BS Repellent, and it is something that I feel that all educators should read. 

We all have our beliefs about educational reform and about strategies that improve student achievement.  Some of these notions are based on experience, others are founded in research, and even more are based on what we tend to believe through our peers and those that we respect.  This is not to de-value the importance of what we have experienced, and in many cases, these experiences are in fact what is borne out by research.  But the question is, how do you know if a claim to the success or failure of an idea or initiative really is true, or simply BS? In times of the shrinking buying power of budgets and increased demands on public education, don’t we think it might be important to do some research into the high-yield strategies that are research proven to work so that we can use those resources as efficiently as we possibly can?  And within that context, I believe that it is important to make sure that we quash some of the BS that tends to cloud our judgment when we are making critical decisions for educational reform and improving student achievement. 

Hattie’s book defines a term called the “hinge point”–this is the point at which it can be said that a factor or innovation has a significant impact on student achievement, and is the average impact of all of the factors/innovations in the six categories of student, home, school, teacher, curricula, and teaching.  The numerical value of this “hinge point” is 0.4.  As a frame of reference, the factor with the highest impact on student had a value of 1.44, and the factor with the lowest (and in fact deleterious) effect was -0.34.

I encourage you to read this book and draw your own conclusions.  However, here is some of the research findings on six interesting educational thoughts that I had, and have now given me pause for thought.  Again, this is all from Hattie (2009), and all credit goes to him and the meta-analyses that he compiled in his book. These are simply a few of my reflections on his book.

1. Class Size

My initial thought: Decreasing Class Size from 25 to 15 could significantly improve student achievement.

The bold, loud claim I hear:  “Decreasing class sizes is a key to student success!”

What the research says:  Of the 138 factors of the meta-analyses done, this was ranked as number 106, and had a impact factor of 0.21, well below the hinge point of showing notable change.  This is based on studies of more than 40000 classes, and nearly 950000 students worldwide. Perhaps not surprisingly, “quality teaching” has nearly double the impact on student achievement than this factor.

My new thought:  Not the high-yield strategy that I believed.

2.  Retaining/failing students

My initial thought:  Holding a student back or failing them is not effective.

The loud, bold claim I hear: “Kids don’t fail enough these days, they just get passed on.  They need to be held back if they don’t have the skills!”

What the research says:  Of the 138 factors, this is ranked 136, and in fact has a negative effect on student achievement at -0.16.  This is based on 207 studies and more than 13000 students worldwide.  This effect gets worse over time, and retained students lose more and more ground on students as the years pass.  Students that are promoted do better on the same outcomes than those who are retained.  As well, those who are retained did poorer across the board, whether it was reading, language arts or math!

My new thought:  The same as my old one, retaining students is not effective, even in a sequentially building course such as math.

3. Ability grouping of students

My initial thought:  Very mixed. It might have some benefits for certain subject areas, maybe math?

The loud, bold claim that I hear:  “Kids need to be grouped by ability so the teacher can focus on their specific needs.”

What the research says:  This factor ranks 121st out of 138, with 500 studies done over hundreds of schools.  It has an average effect of .14, and mostly for higher achieving students.  In specific subject areas, it has almost no impact: in English, .02, and in Math ZERO effect.  But it has “profoundly negative effects” on student’s feelings of equity.

My new thought:  There is no benefit to ‘streaming’ students in schools, and with the changes in the current Math curriculum in British Columbia, the idea of creating adapted classes for students will have more cost than benefit.

4. The relationship between the teacher and the student

My initial thought:  This is a very important factor for student success.

What the research says:  Of the 138 factors, this ranked #11, with an impact factor of .72.  This is based on nearly 230 studies and more than 350000 students.  Some behaviours that are particularly important for teachers are empathy, warmth, and ‘non-directivity’: allowing students more student-initiated or student-directed activities for learning.

My new thought:  This is even more important than I initially believed (and I believed relationships to be tremendously important), and something that I want to emphasize at our school.

5.  The effect of Principals and School Leaders

My initial thought:  This is a very important factor for student success, especially transformational leadership–not because I am a Principal, but because I have read and heard so much about how important school leadership is supposed to be. 

What the research says:  This ranked 74th out of 138 factors, and came from nearly 500 studies spanning over 1.1 million students.  A quote from the book: “Instructional leadership refers to those principals who have their major focus on creating a learning environment free from disruption, a system of clear teaching objectives, and high teacher expectations for teachers and students.  Transformational leadership refers to those principals who engage with their teaching staff new in ways that inspire them to new levels of energy, commitment, and moral purpose to collaboratively overcome challenges and reach ambitious goals…the evidence supports the former (instructional) over the latter (transformational).” (Hattie, 2009).

My new thought: Not quite as important as I thought, and I need to spend more time and focus on setting clear teaching objectives and high standards for my staff and students.

6.  Formative Evaluation of programs

My initial thought:  Extremely important for teachers to adapt and change their methodologies in response to student learning. Using student data to guide instruction and reflection through collaboration with their peers is something that we have been focussing on in our school through our change in structures.

Loud, bold claim I hear:  “I know what works in my class!”

What the research says:  This ranks as #3 of 138, with an effect of 0.9 over nearly 4000 students and 38 studies.  Teachers being purposeful to innovations in that they are looking to see “what works” and “why it works” as well as looking for reasons why students do not do well lead to improvement in instruction and student achievement.

My new thought:  This is the high-yield strategy that can really make a difference at our school, and through the Professional Learning Community Model of providing time for teachers to collaborate and reflect on teaching practices, we have seen a marked increase in the success of our students.

Again, these are just a very few of the highlights of this very important book, and if you are interested in challenging your own beliefs about education with the research that is out there, I think Hattie’s “Visible Learning” is a must-read.  Furthermore, if you are interested in finding out “what works” in terms of high-yield strategies to maximize your staff development, I think this is an excellent beginning. Or, if you just want to be able to call BS on people who make loud, bold statements about education without having done the research, the research here is your BS repellent.

And that is no BS.

Hattie, J., (2009). Visible Learning: A synthesis of over 800 Meta-Analyses relating to achievement. Routledge, Oxon.


  1. Greg Gorman said:

    Outstanding article thank you for sharing. I was like you and thought some of the factors mentioned had a great impact on student learning. It makes me think we are missing the mark with some of our staff development and will have to re-direct our goals.

    January 9, 2011
    • cbirk said:

      I had the same feelings about some of the things that we have looked at in ProD as well, Greg! The one that jumped out of the page at me in the book is about gender…I won’t give it away for you, but it surprised me. This book challenged many of my beliefs, and what I liked about it was that it wasn’t just one person’s individual rants, it was based in hundreds of studies around the world.

      Hope you enjoy it as much as I did!

      January 9, 2011
  2. @matt_gomez said:

    Is there not a correlation between the relationship of student/teacher and class size (#4 and #1)? Even for the best teachers there has to be limit in the amount of quality relationships a teacher can make. Any research on that aspect? My argument for class size has nothing to do with logistics, behavior, classroom management etc. I could function and teach 50 kids if necessary, but the relationships would suffer.

    January 9, 2011
    • cbirk said:

      You bring up a good point, Matt. Not being the researcher, and certainly not the expert, I am guessing that these variables are being considered in singularity. With the example of class size and student-teacher relationships, and argument could be made that simply reducing class size does not necessarily create positive relationships, the teacher does. From my own perspective, I think that educators who truly wish to make positive and effective relationships with their students will make those relationships whether their class size is 15 or 25 (and remember, this was the range, not 50 students–you are right, I think that would be very different). But if a teacher feels that student-teacher relationships are not an important part of their class, having a class size of 15 rather than 25 will make little difference.

      January 9, 2011
  3. Cale:

    This is a very valuable piece: I am indebted to you. I can’t wait to get the book and blog about it myself in the months ahead.

    I had the same reaction to the particulars as Matt (above) did. If relationships are so important, how is it possible that relationships can be as strong in much larger class size dynamics? I realize you can have a small class/low ratio school which makes no priority of such relationships, but if you have two schools side by side, both of which do, wouldn’t you almost certainly be more successful in the smaller environment?

    I do realize I am not at all “arguing” with you yourself, Cale– I am just inquiring into the underlying issues in the research. I will bring these questions, and others, to my own reading of the book: I can’t wait!


    January 9, 2011
    • cbirk said:

      Thanks for the comment, Jonathan. Matt does raise a good point, and I responded to it above with my best guess. However, we all know that education is multivariate, and I think the thrust of this book for me is to try to determine which strategies might have better “bang for the buck” for our situation. The cost of creating classes of 15 from 25 at our school would be incredibly (and I would say ‘prohibitively’) expensive, so it is good to know that there are a number of other innovations that we can try that will have more of an impact on students at a fraction of the cost.

      You will enjoy the book–it took me a bit to wrap my head around the research methodology, but once I did, I really couldn’t put it down.

      January 9, 2011
  4. Aron Campbell said:

    One of the best articles I have read on cpchat…thanks for sharing what seems to be invaluable in our efforts to make the most of what precious time and resources we have. Am ordering immediately.loved the class size research and relationship to quality teaching. Great job!

    January 12, 2011
  5. Nora Perron-Jones said:

    Thank you for this valuable review and wonderful format, sharing what you’ve unlearned/ relearned. It’s hard to find a good read that doesn’t just cite research stats to support an author’s opinion. I look forward to reading this book and wrapping my head around some sound, empirical evidence to rethink my role in student achievement and instructional leadership.

    January 14, 2011
  6. […] Connected Principals: ‘Research: The Educational BS Repellent’ een fascinerende recensie over Visible Learning van John Hattie. Hattie heeft een grote […]

    February 1, 2011
  7. […] John Hattie noemt in zijn grote meta-onderzoek feedback geven één van de meest effectieve onderwijsstrategieën. Welke implicaties heeft dit voor de manier waarop (en waarom!) we leerlingen beoordelen? Een mogelijk antwoord daarop biedt het artikel ‘The Power of Feedback‘ (2007). Hierin beschrijven Hattie en zijn collega Helen Timperley een model voor effectieve feedback. […]

    February 28, 2011
  8. Tom Kenny said:

    I have thoroughly enjoyed reading your work, here.

    March 5, 2011
  9. Katie R said:

    Metadata analysis is all very well, but what we need is more specific research done in the multi-cultural classes of over 30 students, with varying levels of mastery of the instructional language. For example, ESOL and 1st generation students. Also, metadata analysys does not allow for cultural differences in learning styles.
    As any student will tell you, class size matters to them. They know that when they fins something hard, they need a teacher who has time to help them individually.

    December 1, 2011

Comments are closed.