The progress loophole of despair

Is it possible that there are certain situations where a school would benefit more from a pupil actually scoring less on a test? Having spent probably too much time playing with the KS2 pupil ready reckoner tool (click here and click ‘how Ofsted and DfE analyse your data’ to download), it would appear so. But this flies in the face of the core purpose of value added measures – that all progress counts – and the key statement in the DfE’s Primary Accountability guidance: ‘It is important that schools are held to account and given recognition for the progress made by all of their pupils’. So, how can we have a loophole where certain pupils are excluded, even if they’ve taken the test? And what are the implications for future assessment arrangements if this loophole remains open?

Having played with the pupil ready reckoner tool, I’ve found a number of interesting scenarios and they involve pupils assessed as HNM or PKS (pre-key stage, which includes BLW, PKF, PKE and PKG):

1) Pupils assessed as PKS who do not sit the test
These pupils obviously have no test score and are instead assigned a nominal score as defined in the accountability guidance: BLW = 70 points; PKF = 73 points, PKE = 76 points, PKG = 79 points. These points are for progress measures only and are not used in the average scores measure. These pupils will be counted in the denominator for the percentage threshold measures.

2) Pupils assessed as PKS who do sit the test but do not gain enough marks to achieve a scale score
Naively, I assumed that any pupil assessed as PKS would not sit the test because they were working below the level of the test, but that is certainly not the case. There are enough examples in the datasets I’ve seen to suggest it’s actually quite common. In situations where ‘PKS’ pupils sat the test but did not achieve enough marks to be awarded a scale score, they are instead assigned the same nominal scores according to their PKS code as detailed above. In the ready reckoner tool, if ‘no scale score awarded‘ is selected in the first box, users are prompted to ‘Please enter a KS2 Teacher Assessment if the test outcome is B (working below the standard of the test) or if the pupil entered the test but did not achieve a scaled score‘. PKS codes are acceptable values and the nominal score (i.e 70-79) is awarded in place of the scale score. Again, these points are for progress measures only and are not used in the average scores measure. These pupils will also be counted in the denominator for the percentage threshold measures.

3) Pupils assessed as PKS who sit the test and gain enough marks to achieve a scale score
This scenario  is quite simple: the pupil achieves a scale score, which supplants the nominal score assigned to the PKS code; and because the lowest scale score is 80, the pupil makes at least a single point gain in their individual progress score (the highest they can get if they don’t score on the test is 79). However, unlike the nominal scores assigned to PKS codes, this score is also included in the average score, so whilst the school makes an almost negligible gain in terms of progress, its average scores are likely to take a big hit. These pupils are therefore included in progress measures, average scores and percentage threshold measures.

4) Pupils assessed as HNM who achieve a scale score
These pupils are not working below the level of test but are assessed as not having met the expected standard. These pupils take the test and the vast majority achieve a scale score – there is a wide range of scores for HNM pupils with many achieving low scores (i.e. below 90). These scores are used in the progress measures and average scores. This is all fairly normal.

5) Pupils assessed as HNM who do not gain enough marks to achieve a scale score
Here’s the loophole. Unlike, PKS pupils, this group are not assigned a nominal score of any sort. When ‘no scale score awarded‘ is selected in the pupil ready reckoner tool, and HNM is then selected in the teacher assessment box, as prompted, instead of the nominal scores awarded to PKS codes, we are instead confronted with a red box stating ‘pupil excluded’. So a PKS pupil that sits the test but does not achieve enough marks to get a scale score is assigned a nominal score from 70-79, but HNM pupils receive no such score and are excluded. The impact of this, particularly on a small school’s data can be enormous. Those nominal scores for PKS codes can have a major negative impact on progress measures, but if pupils were instead assessed as HNM and had not scored on the test, they’d have been excluded and the negative impact would disappear.

What concerns me is that schools may start to be strategic in their assessments if this loophole remains open. Yesterday I saw an example of a pupil that had been assessed as HNM, who gained just enough marks to achieve the lowest scale score of 80. The pupil ended up with individual progress scores of around -12, which has a profound impact on the school’s overall progress measures. If the pupil had achieved just a few less marks, then no scale score or nominal score would have been awarded and the pupil would have been excluded from the progress measures. The -12 would disappear.

In another school, a PKE pupil had been entered for the test but failed to achieve a scaled score. In this case the nominal score of 76 is awarded, and again there is a big impact on VA due to the large negative deficit. If this pupil had been assessed as HNM then no nominal score would have been awarded, the pupil would be excluded from progress measures and there would be no major, negative impact on the school’s progress measures.

The worst case scenario is that schools assess pupils as HNM rather than PKS and enter them for the test knowing/hoping they don’t score, thus ensuring there is no negative impact. If they do score on the test then at least the score will be higher (i.e. 80+) than the nominal score they would be have been awarded for the PKS code (79 maximum). So, either they score and there is some benefit, or they don’t score and there is an even greater benefit. I’m sure (I hope) this wouldn’t happen but we currently have a ridiculous situation where schools could gain from entering pupils for a test they may not be able to access, award them an incorrect teacher assessment, and hope they don’t score.

The simplest solution is to award a nominal score of 79 points to those ‘HNM’ pupils who take the test but do not achieve enough marks to be awarded a scale score. That would solve this particular problem but considering the big negative deficits associated with PKS nominal scores and the considerable impact these have schools’ progress measures, there are still serious question marks over the whole methodology.