Back in February I wrote this post, speculating on the potential issues of the proposed writing progress measure. That measure is now a reality and the issues predicted in the blog post have taken root. I don’t have a problem with VA measures per se – although I would prefer it to be contextualised – but I do have an issue with VA when it’s based on teacher assessment. The problem is that teacher assessments are too broad, too inconsistent, and too prone to bias when high stakes are involved. I certainly echo the proposals of the Headteachers’ roundtable alternative green paper, which calls for the removal of teacher assessment from high stakes accountability measures. In my blog I discussed the problems of the writing VA measure under levels and suggested that it would only get worse in the post-levels world. To understand this we need to look at how VA works and why writing differs from reading and maths.
VA involves comparing a pupil’s attainment score at KS2 against the national average score for pupils with the same KS1 prior attainment. Many of us will now be familiar with the lookup table on p16-17 of primary accountability document, which shows the average scores in reading, writing and maths for each of the 21 prior attainment groups. These are the benchmarks against which each pupil’s actual KS2 scores in the three subjects are compared this year (please note: they’ll almost certainly increase next year). I don’t have a big problem with the reading and maths benchmarks – they are fine graded, and pupils, having sat tests in those subjects, will have a fine grade to compare it against. In short, the reading and maths benchmarks are achievable.
That is not the case for writing. Again the benchmarks are fine graded but unlike reading and maths, in writing they are unachievable. Because there is no test score in writing the decision was taken to apply a nominal score to the teacher assessment. In my blog, I speculated that 90, 100, and 110 would be possible nominal scores assigned to the WTS, EXS and GDS assessments. We now know that these scores are 91, 103 and 113. This doesn’t really change the problem; if anything it makes it worse. To try to understand this a bit better, I put together the following table, which shows every possible progress score in writing this year for every possible TA outcome from every start point:
The first thing to note – and this is something that confuses quite a few people – is that writing benchmarks in column 3 are fine graded. How can this be when they haven’t sat a test? Well, consider all those scores of 91, 103 and 113 (and the nominal scores attached to pre-key stage judgements – another issue). If we add up all the various nominal scores for all pupils in a particular prior attainment group (thousands of pupils) and divide by the number of pupils in that group, we will of course end up with a fine grade. But as mentioned above, it’s a fine grade that in many cases, due to the scoring system, pupils can’t get close to.
Now have a look at the next few columns. Negative numbers indicate pupils making supposedly less than average progress. It is therefore not possible for a pupil assessed as BLW, PKF or PKE to make more than average progress, and the only PKG pupils that make more than average progress are those that are in prior attainment group 1, i.e. pupils that were below P7 average at KS1. Now take a look at pupils with a KS1 average point score of 10 or more – they need to be at least EXS to make more than average progress. In reading and maths, they just need a few more marks on the test to shift from negative to positive but in writing they need to make the leap from WTS to EXS. For prior attainment group 7 (KS1 APS 10 to <12) this leap changes their progress score from -1.6 to +10.4; and for prior attainment group 9 pupils (2C average at KS1) the progress swings from -5.69 to +6.31 if they are awarded EXS instead of WTS. Meanwhile pupils in prior attainment group 12 (2B average at KS1) assessed as WTS at KS2 end up with a deficit of -9.75. And EXS isn’t good enough for prior attainment group 16 (2A average at KS1). Pupils in this group need to achieve GDS in order to make more than average progress.
This is a measure based on an illogical scoring system applied to unreliable data. The methodology results in wild wings into positive or negative territory, which have a profound impact on overall progress scores, especially in small schools. Consequently, many schools are likely to have one eye on the progress lookup table when making their writing assessments next year. High stakes influencing outcome. Yes, there have been reassuring statements from Ofsted and the DfE regarding this year’s writing teacher assessments, but I can’t see how the writing progress measure is going to improve much next year. This measure is so flawed that one has ask ‘why bother?’ Is it really worth it? Is a writing progress measure really so important that we are willing to sacrifice accuracy and fairness? Not really – it makes no sense – so let’s ditch it.
It’s time to admit that we can’t measure progress in this way.
Subscribe to receive email updates when new blog posts are published.