Standards in mathematics

Another sad reflection on declining standards, this time from an anonymous student.

Mathematical literacy and PISA revisited

Ireland's recent low place in the international "league tables" produced by PISA, especially for maths, has got a lot of attention. This example illustrates the problem nicely.

Ireland's PISA results: myth and reality

The recent results from the PISA surveys have got a lot of attention. In somewhat apocalyptic terms, the Irish Times referred to “2010: the year Irish education fell to earth” and invited reader’s opinions on Ireland’s “Falling literacy levels” though the Irish Independent did carry a more balanced assessment. A careful analysis of the data reveals a more complex picture.

Firstly, these assessments do not show in absolute terms how well or badly our students are doing since in each wave the data are normed to have a mean of 500 and a standard deviation of 100. Comparing the results for 2000 and 2009 will not tell us whether our current 15 year olds know more or less than the equivalent cohort 9 years earlier. It just shows where they are relative to students in other countries. So even if our current 15 year olds have slipped in the rankings, they might actually be better academically than the earlier Irish cohort but have simply been leap-frogged by other countries. Let us not forget that results in public exams have been trending up. Maybe it's grade inflation, maybe not.

Of course rankings do matter, for example foreign investors may only care about the relative skills of countries’ workforce. However when our rankings fall, as they have recently, there may be a tendency to use this as a stick to beat the education system. But it can hardly be to blame if our students are actually learning more but other countries simply got better.

Secondly, PISA includes a diverse range of countries including, for example, Liechtenstein, Kyrgyzstan and Macau. With no disrespect to these, should we really care where we stand relative to them? Does it not make more sense to pay much more attention to countries which are more relevant to us economically and/or are simply bigger?

Thirdly, PISA has changed significantly since it started. In 2000, there were 32, largely OECD countries. This grew to 41 in 2003 and in 2006, which focused on science there were 57 countries. The latest wave has 64 countries - some are actually territories. So when you look at a ranking, you are counting how far you are from the top. But one could equally look at how far one is from the bottom. Now as other countries join, it’s likely some will come in above us and some below. So the ranking can be misleading. Looking at how far we are from the bottom might tell a different story. A fairer comparison over time would just compare us with those countries that were in with us from the beginning and indeed some commentators have looked at our rankings just with the OECD.

To look closer at this consider the reading results. In 2000, PISA put our students between 3^rd and 9th (out of 32). For statistical reasons, they didn’t give exact rankings but let’s take 6^th place as an average. That puts us in about the top 20% of countries. By 2009 our ranking had slipped to 21^st. That may seem a precipitous decline but there are twice as many countries in the frame: we are now only in about the top one third. So while it is nothing to be happy about, it’s not quite the disaster it appears at first blush. Mathematical skills are often seen as being particularly important, so where do we stand? In 2000 we were ranked between 16^th and 19^th , in around the top 55% of countries. In 2009 that had fallen to 20^th but given that there are more countries we are just about in the top half: arguably this is an improvement. Looking at the science results tells a similar story.

So the point is that one has to be careful to jumping to conclusions about how our students and schools are doing from simplistic comparisons. When one studies the detailed reports produced by the OECD it becomes obvious that there is far more information than hits the headlines. The research is particularly informative on issues of socio-economic disadvantage. In all countries, those students from a better-off background do better but the extent to which this is the case differs markedly. In a fascinating analysis, the OECD considers what it defines as “resilient” students, these are students who are in the bottom quarter on the socio-economic scale but who perform in the top quarter in the assessments. It is a good summary of the extent to which young people are not held back by their background. So who tops this particular league table? It is dominated by the Far East, the top 7 are (in order) Shanghai, Hong Kong, Korea, Macau, Singapore, Finland and Japan. And us ? Ireland is pretty average, ahead of the US, the UK and many European countries though not by much. That we are way ahead of Kazakhstan, Panama or Dubai in this respect seems scant consolation.

Anger, we have been advised, does not constitute a policy. Neither do simplistic statistics or headlines. Ireland faces particular challenges in adapting its educational system to changing needs and technologies, a changing international economy and a much less favourable fiscal situation. It is imperative then that we carefully monitor the extent to which educational resources produces results, how we fare relative to our competitors and the extent to which our schools mitigate or reproduce social inequities.

The news from PISA

The latest results from the Program for International Student Assessment (PISA) are out and Ireland does not do too well. Sean Flynn has a fairly trenchant take on it. While I don't necessarily agree with everything he says, he makes some good points. The bottom line seems to be that we have slipped down the international league table significantly using the scores. PISA measures attainment of 15 year olds, usually in reading, science and mathematics. There have been several waves of data so far.
When the issue of "school league tables" comes up, many in the education sector are quick to oppose them on the basis that they don't provide a balanced comparison, not comparing like with like etc. Well if you believe that, you can't then use these "international league tables" to rank countries with different education systems, different cultures, different curricula and so on as you are certainly not comparing like with like either.
What I am unclear about is whether PISA only tells us where we are relative to others. That is, say attainment in Ireland has not changed in absolute terms but other countries have simply got better than us, then will we still move down the table? My guess is Yes but in reality its probably a bit of both going on: that is attainment may have fallen in Ireland and we may have been leapfroged by others also. My recollection is that PISA scores are normed to have an overall mean of 500 so it can't tell you about absolute changes but I may be wrong.

Class-size effects in PISA : further results

Further to my recent post, I show a somewhat detailed analysis above. The dependent variable is the students reading score which is an arbitrary function of class-size (i.e. in their English class, since this is data for Ireland) and of the teacher/pupil ratio in the school. In addition, I control linearly for the students sex, their age in months (they are all about 15), their birth order, their fathers education, school type (private/public) and a few measures of the home environment (like number of books). These controls don't actually make much difference.
So again we see a positive gradient with students in bigger classes doing better though it seems flatter than previously shown. The teacher/pupil ratio doesn't matter: using the absolute number of pupils gives the same result i.e. there are no economies or diseconomies of scale. Repeating the analysis for the maths score gives much the same results. The positive gradient is a puzzle though this is not the first time it has been found.

Some unpleasant educational arithmetic

Given the major pressures on the public finances and hence on public spending on education, the question of class-size is likely to rear its ugly head again. We don't have good evidence for Ireland though there is some good potentially data. So what does it tell us - at least at first blush?
Using PISA data for Ireland I show a kernel regression of the students (15 year olds) against class-size. The results show that as you lower class size from 40 to about 35 there is a higher score- as you might expect. After that however, its down-hill all the way: smaller classes are associated with worse scores. This is actually robust to inclusion of school effects and various other controls. In work in progress with a colleague it even seems to be robust to controling for endogeneity (using two distinct identification strategies).