The Nuclear Option

I guess we’ve been building up to this. As Sean Harford announced at a recent conference that, under the new Ofsted framework, inspectors will “ignore all tracking data”, delegates experienced a collective jaw-drop moment. We sat up like Meerkats, alert, turning to one another in bewilderment with silent questions on our lips: “what the….?”.

(This proposal has now been confirmed with more detail in a recent Ofsted video.)

My initial reaction was “Yes!” and that was the overwhelming reaction on twitter, too. For the last four years I’ve been tweeting, blogging and speaking about the reinvention of levels and the lengths schools have gone to maintain the status quo whilst attempting to convince themselves and others that what they are now doing is something new. Something not levels. But if anything, the approaches commonly employed by schools now are much worse than those we had before: sublevels have been split and split again to the point that some schools have decimal point systems whereby pupils supposedly progress from 5.0 to 5.9 across year 5, and receive labels such as ARE-1, Developing-, or Exceeding++. I listen to people trying to explain (and justify) this stuff, and wonder if they ever have an out of body experience. Do they ever stop and ask themselves “does this make any sense?”. It makes no sense at all to me, so how can it possibly mean anything to an inspector especially considering the narrow window of opportunity that a senior leader has to explain their approach within the time constraints of an inspection?

For years inspections have revolved around data: statutory, end of key stage data that shows how well one (often small) cohort did on one particular day; and internal tracking data that was highly subjective and ropey to say the least. And yet, because of levels that provided an illusion of comparability, we had a form of common currency. We all bought into the ‘truths’ of expected progress – 3 points per year in key stage 2; 5 points per year at key stage 1 – and no one ever really questioned it. We went along with the collective illusion because we were in too deep, and it was all too convenient. But it was smoke and mirrors. I remember a head telling me how he changed a formula in a spreadsheet the night before an inspection. Previously it had converted all those that achieved a good level of development at EYFS to 7 points (1C). Realising that this meant pupils only making 8 points of progress if they got a 2b at KS1, he decided to change it so they were assigned a baseline score of 3 points instead; and now they made 12 points of progress across KS1. The inspector took a look at the data the next day and announced that “5 points is expected and your pupils are making 6 on average, so that’s above expected.” Extraordinary. And the same is happening now: schools invent more and more steps and bands to ‘prove’ progress, but it proves nothing beyond the fact that the school has invented more categories separated by more arbitrary thresholds, and teachers have to spend more time working out where to place pupils. It is all part of a huge game to convince people – often external agencies – that pupils are making progress. 

And now Ofsted are about to press the reset button.

The reasons for this are threefold:

1) Data is a distraction. Most inspections are short and time taken to explain convoluted tracking systems is time taken away from more informative activities like observing lessons, looking at pupils work, and talking to teachers and pupils.

2) Data is unreliable. I’ve certainly heard some horror stories. Let’s face it, basing any judgement of a school on subjective data that the school has total control over is just asking for trouble. Someone said to me recently that inspectors should only resort to the tracking data if they feel that standards are poor and something is amiss. Really? How can the data be relied upon under such circumstances?

3) Data is too diverse. Too many systems, too many approaches. Different scales and classifications; points, bands and flight paths; three, six or nine steps per year; pluses and minuses; RAG-rating and purple zones; point-in-time assessment or levels-style best-fit; expected or more than expected progress; emerging or beginning or exploring; standardised tests or teacher assessment. A bewildering cacophony of data noise. Sometimes you need to put the ear defenders on. 

The writing has been on the wall for sometime, the tone being set by the myth-busting section of the Ofsted handbook and more recently by Amanda Spielman, HMCI, in key speeches:

We do not expect to see 6 week tracking of pupil progress and vast elaborate spreadsheets. What I want school leaders to discuss with our inspectors is what they expect pupils to know by certain points in their life, and how they know they know it. And crucially, what the school does when it finds out they don’t! These conversations are much more constructive than inventing byzantine number systems which, let’s be honest, can often be meaningless.”

Essentially, Ofsted have woken up to the fact that much of the data they have come to rely on over the years may well be a pile of fabricated nonsense, and their only option – the nuclear option – is to ignore it.

Those schools with effective systems in place will carry on doing what they’re doing – they’re not doing it for Ofsted anyway – but clearly not all schools are happy about this proposal. There are those that have invested heavily in tracking systems and implemented them almost entirely with Ofsted in mind only to be told that inspectors will ignore it. My advice to those schools is to re-evaluate the purpose of tracking and re-engineer systems from classroom upwards, focussing on workload and collecting only what is needed to support teaching and learning. No teacher ever buys into a system designed around an accountability cycle anyway. Systems should be simple; systems should be customisable; systems should make sense. Get it right and a healthier data culture will emerge.

But there are also those schools that use standardised tests for both formative and summative purposes, that provide robust indicators of attainment and good evidence of progress, as well as revealing gaps in learning. Some of these tests also have a strong correlation with KS2 results enabling schools to predict outcomes with a high degree of accuracy. They maintain that the data from these assessments are accurate, valid and reliable. The numbers are not made up, they tell you something. The nuclear option has collateral damage.

I suppose Ofsted could argue that if systems are effective then the impact will be evident in pupils work and in conversations with teachers about learning so they don’t need to look at the output. But how many books can an inspector look at over the course of a day? Should we wilfully ignore high quality data because a lot of tracking data is unreliable? Are we throwing the baby out with the bathwater?

There is one statement at the end of the video that suggests all is not lost for those schools that have faith in the usefulness of their data and feel it should be considered during an inspection: 

Our intention as we head towards the new inspection framework is to consider data more in the context of the quality of education, that schools are providing children and young people.”

And what of those schools that stubbornly persist with the pseudo-science of levels in a post-levels world? I recall a post-Ofsted conversation with a headteacher who, having shared pages of tracking data and progress measures with the inspector, was asked a very simple question:

Can you explain to me, what is a point of progress?”

She admitted that she did not know how to answer. But the next question was the killer:

If you can’t explain what it means, what impact can it have?”

For years, this data has been recorded, aggregated, analysed, shared, and pored over. Subjective measures have been treated like SI units, as if the product of highly calibrated machines, with no one ever stopping to ask ‘what does it mean?’

The Making Data Work report makes this recommendation:

Ofsted inspectors should ask questions about whether schools’ attainment data collections are proportionate, represent an efficient use of school resources, and are sustainable for staff

Ignoring tracking data probably won’t stop bad practice, but inspectors asking probing questions about the reliability, meaning and impact of data, and the workload required to produce it, could kill high cost, low benefit approaches within a year. 

And maybe they wouldn’t have to deploy the nuclear option after all.

Subscribe to receive email updates when new blog posts are published.

Share this article

1 Comment

  1. […] Source: The Nuclear Option – Sig+ for School Data […]

Leave a Comment