Bad science: Are your well-being questionnaires getting to your pupils’ “backstage?”

I was lucky enough to teach Biology and Science for some years and I enjoyed delivering the vast majority of the curriculum to my pupils.  However, the bit I dreaded was preparing the pupils for their coursework assignments.  It required teaching them about scientific conventions and protocols like the meaning of the words ‘validity’, ‘reliability’, etc (AQA Science GCSE Subject Vocab)

The pupils found this challenging and I was desperate for an analogy to help break it down.  One of the nation’s ‘red top’ newspapers obliged me one day.  The article was bemoaning that the UK’s teachers were apparently the worst in the world.  This was a headline I couldn’t easily ignore.  Maybe they are right but I would love to test the science behind this.  A budding journalist approached members of the public with 10 random questions and recorded their answers.  By random – I mean random.  It was a list of questions akin to the Citizenship Test (British Citizenship Test) – one on geography, one on history, science, maths and so on.

Amongst the number the reporter managed to convince to take part were about 6 or 7 (probably unwitting) teachers out shopping at the weekend.  The newspaper chose to focus on these 7 alone and rank them by their scores out of 10.  Oxford Street is a fairly metropolitan and international destination for shopping so the list (by pure chance) did include teachers from a range of countries.  The one teacher from England scored something like 3/10 and ranked bottom of the seven.  On this basis, the headline was “Teachers from England are the Worst in the World.”  As a science teacher, I saw what I was after to help my pupils understand valid and reliable scientific processes.

I recounted this tale to my pupils in my next lesson with them.  Then I asked them to get into groups with the article cutting in front of them and pull apart this hypothesis.  They very quickly came up with the following:

• The population size was too small to make a valid conclusion;
• The population was not representative of the entire population of teachers;
• The participants were not aware of the rationale behind the questions;
• The participants had no time to prepare;
• The questions did not test teaching ability;
• There were too few questions;
• The questions did not focus on a teacher’s specialist subject;

It was a great lesson in critical thinking and I constantly encouraged my pupils to vigorously evaluate any ‘scientific’ claim they came across.  They would regularly bring me other ‘scientific’ claims they found in the media.  Good for them!

STEER Education talks about a child’s ‘backstage’ and their ‘front stage.’  Just like a theatre, all humans present a front stage to their friends, family and publically.  This can be very different from what they are thinking privately on their backstage.  Many schools, and indeed inspectorates, (OfSTED pupil survey) (Estyn pupil survey) use questionnaires to gather pupils’ voices.  I have no doubt that these give some valid data on what the pupils might be thinking and feeling.  But what about those who:

1. Don’t want to answer what they are really thinking?
2. Don’t know how to ‘measure’ what they are being asked for e.g. ‘feeling safe’
3. Don’t know if they are ‘safe’ or not?
4. Can’t distinguish between “feeling” safe and BEING safe?
5. Don’t distinguish between evidence as the basis for their answer that may have happened yesterday or 5 years ago?

STEER calls these pupils the “Hidden Middle” – are you certain that you are identifying these pupils with your current pupil voice tools?

I know as an ex-inspector when, during an inspection, I chatted to pupils about their answers to these surveys – one pupil’s interpretation of ‘unsafe’ was being made nervous by the unruly horses in the field next door to the school.  Compare this with another child’s interpretation of fear of being mugged on the way home.  Both scenarios were ‘scoring’ equally on the survey.  I know as a father when I took my own children walking along the “Edges” in Derbyshire – I would watch them dancing around on the cliff edges with not a fear in the world whereas I was almost sick with terror.  Their assessment of their safety was poles apart from mine.

My advice: always be prepared to challenge the limits of the evidence being presented to you and try to dig deeper behind the headlines.

STEER Education offers an online platform to schools which alerts them to students who may have emerging mental health risks but are not showing visible signs of vulnerability.  Steer Tracking provides a 10 min online assessment accessible from Y3-13, completed by all students 2 or 3 times a year.  Schools have their own online platform to manage the assessments and view their student’s data, and bespoke targeted guidance for teachers to signpost every student and reduce their risks is provided.  School Group leaders have a termly Executive Summary Report (the dashboard you need) to ensure there is a comprehensive overview of all of your schools.