Evaluating State Programmes - “Natural Experiments” and Propensity Scores

Evaluating State Programmes - “Natural Experiments” and Propensity Scores

Denis Conniffe
Vanessa Gash
Philip J. O'Connell

The Economic and Social Review, Vol. 31, No. 4, October, 2000, pp. 283-308

Abstract
Evaluations of programmes — for example, labour market interventions such as employment schemes and training courses — usually involve comparison of the performance of a treatment group (recipients of the programme) with a control group (non-recipients) as regards some response (gaining employment, for example). But the ideal of randomisation of individuals to groups is rarely possible in the social sciences and there may be substantial differences between groups in the distributions of individual characteristics that can affect response. Past practice in economics has been to try to use multiple regression models to adjust away the differences in observed characteristics, while also testing for sample selection bias. The Propensity Score approach, which is widely applied in epidemiology and related fields, focuses on the idea that “matching” individuals in the groups should be compared. The appropriate matching measure is usually taken to be the prior probability of programme participation. This paper describes the key ideas of the Propensity Score method and illustrates its application by reanalysis of some Irish data on training courses.

http://www.esr.ie/vol31_4/1Conniffe.pdf

"Key to boosting job hopes lies in training and placement"

As part of an Irish Times series on the unemployment crisis, equally neglected by government and commentators, the Geary Institute's Liam Delaney discusses the contribution of active labour market policies.

Some evidence on Active Labour Market Policies

In the (perhaps unlikely) event that policy makers take an interest in research on active labour market policies with a view to actually doing something sensible about the crisis in Ireland, this might be of interest:

Does subsidised temporary employment get the unemployed back to work? An econometric analysis of two different schemes
Gerfin, M; Lechner, M; Steiger, H
Subsidised employment is an important tool of active labour market policies to improve the reemployment chances of the unemployed. Using unusually informative individual data from administrative records, we investigate the effects of two different schemes of subsidised temporary employment implemented in Switzerland: non-profit employment programmes (EP) and a subsidy for temporary jobs (TEMP) in private and public firms. Econometric matching methods show that TEMP is more successful than EP in getting the unemployed back to work. Compared to not participating in any programme, EP and TEMP are ineffective for unemployed who find jobs easily anyway or have a short unemployment spell. For potential and actual long-term unemployed, both programmes may have positive effects, but the effect of TEMP is larger
Labour Economics (2005) 12(6), 807-835

No fancy econometrics needed

The evaluation of labour market treatment effects is one of the most actively researched areas in applied econometrics. Getting good estimates of policy relevant parameters presents some quite technically challenging issues.
But sometimes you really don't need all that. The story below from yesterday's Sunday Business Post gave quite a staggering example of this. A program to get the long term unemployed back to work cost €39m with 46 participants (i.e. almost €900k each) and apparently did not get any of them back to work.
The article adds "Forfas said the state was spending €970 million on a number of jobs and training programmes, but found that there was ‘‘significant information deficit’’, in terms of being able to measure their ‘efficiency and effectiveness’ ".
Indeed. The resources necessary to do a first class evaluation of such programs would be trivial by comparison.

http://www.sbpost.ie/news/failed-work-programme-cost-state-almost-900k-per-person-48270.html

Evaluating State Programmes: “Natural Experiments” and Propensity Scores

DENIS CONNIFFE, VANESSA GASH, PHILIP J. O’CONNELL

The Economic and Social Review, Vol. 31, No. 4, October, 2000, pp. 283-308

Abstract: Evaluations of programmes — for example, labour market interventions such as employment schemes and training courses — usually involve comparison of the performance of a treatment group (recipients of the programme) with a control group (non-recipients) as regards some response (gaining employment, for example). But the ideal of randomisation of individuals to groups is rarely possible in the social sciences and there may be substantial differences between groups in the distributions of individual characteristics that can affect response. Past practice in economics has been to try to use multiple regression models to adjust away the differences in observed characteristics, while also testing for sample selection bias. The Propensity Score approach, which is widely applied in epidemiology and related fields, focuses on the idea that “matching” individuals in the groups should be compared. The appropriate matching measure is usually taken to be the prior probability of programme participation. This paper describes the key ideas of the Propensity Score method and illustrates its application by reanalysis of some Irish data on training courses.