Thursday, March 04, 2010

Grades

1. How are college-grades determined? The economics literature points towards a simple answer: "by spending more time on study". I don't provide links here, but for evidence on the positive relationship between study-time and grades, see Arulampalam, Naylor and Smith (2007), Stinebrickner and Stinebrickner (2007) and Martins and Walker (2006). Dustmann and Van Soest (2007) show that part-time employment while in school has a negative effect on males' exam performance. Kalenkoski and Pabilonia (2010), DeSimone (2008), Joensen (2007), Oettinger (2005), Van Dyke et al. (2005), Stinebrickner and Stinebrickner (2003) and Ehrenberg (1987) all provide evidence that working (to varying degrees) while in college has a harmful effect on grades.

2. Why do students care about their grades? Again, I don't provide links here, but for evidence on the positive relationship between college grades and subsequent earnings, see McIntosh (2006), Loury and Garman (1995), Jones and Jackson (1990), Filer (1983) and Wise (1975).

3. What about grade inflation? Some economists are concerned about its interaction with the signalling model of labour market earnings. Others extend this concern to how social background may end up as a driver of labour market sorting.

4. However, grade inflation is hard to measure. In the paper 'More For Less or Less for More?', Geraint Johnes (Lancaster) and Robert McNabb (Cardiff) emphasise that studies of grade inflation have not distinguished between the effects of declining standards and increasing efficiency. "Yet both of these phenomena can plausibly explain an improvement in measured performance of educational institutions over time." Johnes and McNabb propose a method whereby competing effects can be disentangled. Their findings suggest that there has been no decline in standards in the UK university sector during the last decade. Efficiency, meanwhile, has risen steadily over the last 25 years.

5. It is also worth bearing in mind that sometimes universities split into high-grading and low-grading departments. Sabit and Wakeman-Linn (Journal of Economic Perspectives, albeit as far back as 1991) discuss the situation in the United States: "Economics, along with Chemistry and Math, tends to be low-grading. Art, English, Philosophy, Psychology and Political Science tends to be high-grading." Interestingly, it appears that there are higher returns to more harshly marked disciplines; see "Ability Sorting and the Returns to College Major" (Arcidiacono, Journal of Econometrics, 2003). "Large monetary premiums exist for choosing natural science and business majors even after controlling for selection."

16 comments:

Mark McG said...

Thanks Martin, I look forward to the report on grade inflation being published in full. All the media stories today focused on the fact that the proportion of Firsts in 3rd level and ABs in 2nd level has risen substantially, but that's hardly proof of grade inflation in and of itself so I hope there is more to the report than just this

Martin Ryan said...

I agree that a more nuanced approach is required Mark. Geraint Johnes and Robert McNabb do the most sophisticated analysis that I have yet seen. They mention how statistical studies of grade
inflation have "typically adopted a rather simple approach in which grade point averages at university are regressed against students’ prior achievement in scholastic
aptitude tests (SAT)."

Without controlling for some measure of prior ability, there are obvious concerns. Johnes and McNabb go further and state that "an obvious problem with this (the above) approach is the assumption that the efficiency
with which higher education converts its inputs (the student intake) into outputs
(university graduates) has remained unchanged over the years."
They use stochastic frontier analysis to address the efficiency issue.

Of course, there are many other nuances: technology has exploded in recent decades. What are the effects of word-processing, spell-checking, the internet, search engines, broadband quality connections, digitisation of academic materials and so on?

There are also institutional changes: more continuous assessment, more 3-year bachelors degrees (in keeping with the Bologna Process) and more semesterisation (also in in keeping with the Bologna Process).

Finally, there is the behavioural economics angle: with more student going to college than ever before (155,100 enrolled in Ireland this year) how do students differentiate themselves from each other? How do they strive towards establishing a competitive advantage in the ever-more saturated graduate labour market. perhaps by trying to get better grades...

Kevin Denny said...

Its also interesting that concerns about grade inflation focus on the top tail but it can also happen at the bottom: fewer fails, pass grades etc.
If a department or school etc fails a lot of students this tends to go down badly. So there is an incentive not to.
Its also worth noting that in the past UCD and other universities had a separate "pass degree". This was abolished in the late 80s or early 90's I think in UCD on the basis that "all our students were really honours students". I suspect it was for resource reasons. The consequence is that there are a lot of people doing honours degrees who would not be otherwise.

Ferg said...

Does anyone remember preparing for leaving cert honours maths? without trying to jump on the bandwagon, it was blindingly obvious that the papers in the late 80s early 90S were far more difficult than the papers we (when I say "we", I mean us whippersnapper 20somethings) sat. While grade inflation might not be the same, and this is one anecdote, I find it hard to believe that the system has not been "bumed down" to some degree.

Peter Carney said...

Can't help but wonder, given the increased number of octogenarians, if its time to start questioning Irish hospitals about their mortality standards?

Much of the media debate is akin.

I thought the University Presidents were fairly comprehensive in highlighting the multitude of factors at play over the past 15 years. They point out improvements in technology, communication, and access to materials, i.e., the Internet, which the media analysis didn't consider. The Presidents also mention Mature students; which I is interesting. It's true that they weren't there in such numbers before our time and they were all...well...very ambitious, shall we say.

Martin Ryan said...

Fergal,

I take your point about exam (and syllabus) content. Geraint Johnes and Robert McNabb consider this when they outline the three main approaches that are possible when investigating grade inflation. For practical reasons, most analysts make do with a statistical analysis of exit grades.

The main point of this post is to flag how naive some of these statistical analyses can be. Also, to encourage highly-skilled graduates who got their qualifications this decade to have pride in their achievements and confidence in their credentials.

I think these graduates should in fact react strongly to the allegation of "dumbing down". There are many reasons to believe why grades should actually have been increasing over the last decade or two. As outlined above, these include better technology, changing institutions and more intense competition between students.

There are arguments in favour of grade inflation too. You raise one Fergal, and another that was emailed to me focused on increasing student demand for mark-ups. I guess there may be a mixture of things happening. I just think it is a bit much to attribute all of the rise in grades to inflation. And most discussion on the issue has gone that way.

Liam Delaney said...

Martin, you mentioned the Flynn effect, whereby people get better at tests over time. Have you found anything on this with relation to grade inflation?

Martin Ryan said...

Kevin,

You raise some interesting points about the incentives of higher education institutions. You're better placed to understand these than I am; but I do appreciate that there would be value in investigating the full distribution. Do we expect the cohort percentage of failing grades to be decreasing over the last couple of decades? Though again, we have to grapple with secular changes in technology, institutions and student competition.

Martin Ryan said...

Liam,

A preliminary investigation doesn't produce too much on how a Flynn Effect may be driving grade inflation. It's certainly salient though, given:

(i) massive efforts to predict likely exam questions by looking at previous exams
(ii) exacerbation of this approach to exam preparation in the highly competitive CAO points system, ever-growing grind-school phenomenon, and frenzy over second-level league tables (based on whether they send students to college)
(iii) wider distribution of Departmental marking schemes and commercially produced student materials that provide "answers to previous exam questions" and more "sample questions".

@ Peter,

Good point about mature students. There is also the possibility that better practices at second-level (or just the increased competition at second-level, as outlined above) mean that there have been more students with high expectations and drive coming onto higher education campuses in recent years.

eoin said...

Peter - I don't even think the Presidents did a decent job of arguing that point. The NUIM President was on Sean O'Rourke pedalling the they-work-harder-that-we-did line, the limpness of which only serves to fuel the 'falling standards' point.

An especially disconcerting aspect of this debate (notwithstanding the government reaction to mild pressure from a couple of multinationals) is the hopelessly uninformed media analysis that it has spawned. It is clear that very few OpEd writers (Madam included) had mobile phones or internet access while in college.

Liam Delaney said...

More international students also.

Keith said...

UCD changed the degree classification criteria two years ago. Previously, penultimate year grades were weighted as 30% and final year as 70%. Now it's 50-50. I wonder if this suppressed (or increased?) the number of firsts awarded last year.

One can achieve a 2.I in penultimate year and it might be an impossibility to achieve a I.I overall, even with 12 A+ grades in final year, under this system.

Martin Ryan said...

Keith,

Those changes may make it harder to get a 1st, but I wonder if they make it harder to get a 2:1?

Eoin,

In relation to the disconcerting Government reaction to mild pressure from a couple of multinationals, its worth reflecting on:

(i) Is industry right?
(ii) What are industry's incentives?

In relation to the first point, evidence from the World Economic Forum's 2009-2010 Global Competitiveness Report ranked the quality of Ireland's secondary and tertiary educational system, as assessed by the business community, at eight out of 133 economies.

"This is further reinforced by the ratings given to the Irish universities by a broad sample of employers and which have fed into the improved performance of the universities in the Times Higher/QS University rankings," IUA chief executive Ned Costello said.

http://bit.ly/d8xFo6

One could retailate with "Of course Ned Costello would say that". But by similar token, what do we expect industry to say? Might certain industries say that Irish graduates are not adequately trained? Might certain industries have incentives to:

(a) pay graduates less than graduates expected during the Celtic Tiger years
(b) encourage the Govt. to sweeten compensating conditions for them i.e. keep corporate tax low
(c) encourage the Govt. to spend more money on R&D

What we do know is that it's tough out there in the graduate labour market. 1 in 3 men under the age of 25 are currently unemployed, and that doesn't include those who are under-employed or working for a low salary. With morale undoubtedly low, the last thing graduates need is for their grade-achievements to be undermined.

Martin Ryan said...

The university presidents have outlined a list of reasons as to why there might be increases in higher awards over time. The IUA statement is available here:

http://bit.ly/drD04Z

This contrasts strongly with the simplistic headline on the front page of the Irish Times Weekend Review, which is:

"How Ireland Dumbed Down"

Did it never occur to some commentators that the Leaving Cert. (and college exams) can be quite a stressful and intense experience for many students? Also, it can be argued that students care much more about these exams than previous generations; and that there is much more pressure involved than there was even a few decades ago.

Keith said...

Martin,

It depends, I guess. Given a poor performance in penultimate year, 50-50 would make it more difficult to move up a degree class. If you invert this, then a good performance in penultimate year can more easily offset a disaster in final year, which makes the 2.I more attainable; especially if you assume that easier modules are taken in penultimate year. I suppose this somewhat comes down to students discounting the future.

I wonder what some of the lecturers at a, eh, more advanced stage of life think about standards: is your average econ more attuned to the material than 15 years ago? Granted, we're aided by such wonderful inventions as BlackBoard, and greater use of PowerPoint/LaTeX for presenting technical material.

Looking back on the quants exams of 2001, they actually look easier than what we're taught today.

Martin Ryan said...

Keith,

It's also interesting to observe that the four constituent universities of the (now historical) National University of Ireland had different grading bands prior to 2002.

2:2's were only given to 55+. 2:1's were only given to 62+.

http://bit.ly/9bcrja