24 August 2009
A-Levels Worse Than Useless
The A-level results came out last week and generated headlines such as, "A-level results: another great leap forward." In the article Ceri Edwards explains,
"Record A-level results! Another year group of teenagers becomes inexorably brighter! Soon they will be building their own space shuttles, ending poverty and cracking world peace, all in three modules!
I don’t mean to be dismissive of any individual’s success, but it is hard to react with anything other than cynicism to a system which gives over a quarter of A-level papers an A grade, with overall pass rates rising for a 27th consecutive year."
According to the BBC broadcast journalist, Tim Hartford, there is a
"1 in 100 million chance of 27 consecutive years of improvement if teaching and examination standards were varying by chance."†
This can be found on the Radio 4 programme, "More or Less, 21.08.2009. There is a listen again facility but this only lasts for seven days - the A-level results section starts just after 20 minutes.
Hartford finds further authority for explaining why this A-level rise isn't just by chance: he talks to Prof Gordon Stanley. Stanley explains that across all of the A-level subjects there is the same rate of increase of exam results year on year (ca 1 - 2 %). But this isn't seen in other education systems at which he looked (Australia and Scotland); he concludes that the increase in results is a function of the standard setting system. Although not directly referenced in the programme, it appears that the paper, Stanley 2008, is being discussed.
Hartford goes on to explain about the switch, in 1987, from normative assessment to criteria assessment. In normative assessment the results from the exam in question for the whole year was collated. The data was analysed such that a grade could be found where 10 % of pupils out of the cohort got a grade A. So, if one year 10 % of the pupils achieved 70 marks (or above) out of a hundred then the grade A mark for that year was 70 marks; if 10 % of pupils achieved 98 marks or above out of a hundred then the grade A cut off for that year was 98 out of 100. Under a normative system; if you got a grade A you were part of the top 10 % of the country for that year. This is in contrast to a criteria assessed system. In this system criteria are set before hand; eg, you need to get 80 marks out of 100 in order to get a grade A. The latter is the system that we are under now.
There is a lot more to the normative vs criteria assessment argument. The broadcast journalist said that in the normative system the A grade was always 10 % of the highest performing cohort: this wasn't true. Every year a committee would sit and fiddle with the figures. It was a numerous clausus mechanism that ensured sufficient numbers of private school pupils got a place at University. While the criticism of the criteria assessed system depends upon an objective measure of what is an A grade? Who's to say? How is it justified? How is it maintained? In fact, all of the criticisms that we are hearing now.
Returning to the Hartford broadcast, it was explained that in maths pre 1987 10 % of students got an A grade but now, the number who get an A grade is 45.2 %. Hartford then invites Prof Timms to discuss some of his work. From 1988, Timms explained that every year he has tested the underlying abilities of A-level students and has compared this ability to the pupils's A-level grade. He found that a comparison of the students over time who had the same underlying aptitudes, had A-level grades that had been rising over time. The grade drift in maths was found to be about one grade per decade with a three grade drift from 1988 to the present. Hartford got a maths E grade in 1992; Timms concurred that this would be the equivalent of a C/B grade now.
The A-levels are no longer A-levels. They have changed in everything but name from 1987 and the objective measure of the neu A-level is changing every year.
This brings us to the title of this post. A-levels are worse than useless because if anyone attempts to use them to discrimate between people of differing ages there is a very good chance that they would fall foul of the age discrimination regulations. Imagine if Hartford saw a job advertised that required a grade C in A-level mathematics; what would Hartford's lawyers make of that?
See, Age Discrimination and Exam Results for an explanation and links.
Stanley, G & Tognolini, J. (2008) Performance with respect to standards in public examinations. 34 September 2008.
† Although Hartford didn't explain how he got this figure, I think he is approximating two to the power of 27. He seems to be asking every year whether or not there will be an increase; if the answer is yes for twenty seven successive years the chances of that occurring are 1 in one hundred thirty-four million two hundred seventeen thousand seven hundred twenty-eight times.