Evaluating State Programmes - “Natural Experiments” and Propensity Scores

Evaluating State Programmes - “Natural Experiments” and Propensity Scores

Denis Conniffe
Vanessa Gash
Philip J. O'Connell

The Economic and Social Review, Vol. 31, No. 4, October, 2000, pp. 283-308

Abstract
Evaluations of programmes — for example, labour market interventions such as employment schemes and training courses — usually involve comparison of the performance of a treatment group (recipients of the programme) with a control group (non-recipients) as regards some response (gaining employment, for example). But the ideal of randomisation of individuals to groups is rarely possible in the social sciences and there may be substantial differences between groups in the distributions of individual characteristics that can affect response. Past practice in economics has been to try to use multiple regression models to adjust away the differences in observed characteristics, while also testing for sample selection bias. The Propensity Score approach, which is widely applied in epidemiology and related fields, focuses on the idea that “matching” individuals in the groups should be compared. The appropriate matching measure is usually taken to be the prior probability of programme participation. This paper describes the key ideas of the Propensity Score method and illustrates its application by reanalysis of some Irish data on training courses.

http://www.esr.ie/vol31_4/1Conniffe.pdf

Stata resources for treatment effects

There are a large number of resources within Stata for the estimation of treatment effects. Some are part of official Stata and others are user written that can be easily downloaded.

To estimate regression discontinuity models, there is a download rd due to Austin Nichols. Further details at Nichols, Austin. 2007. "Causal Inference with Observational Data." Prepublication draft available at http://pped.org/stata/ciwod.pdf. It is published in the Stata Journal now I think.

To estimate IV models there are several options in Stata.
ivregress is the main program. A download ivreg2 due to Baum, Schaffer & Stillman is very useful - I recommend it. Make sure you get the latest version. Their paper should be used in conjunction with it: http://ideas.repec.org/a/tsj/stataj/v7y2007i4p465-506.html. xtivreg2 is the equivalent program for panel data.

ivtobit and ivprobit do what their names suggest. If using them, you need to satisfy yourself that they are consistent estimators. Caution is appropriate where the instrumented variable is binary. In the latter case biprobit may be better.

cmp (due to David Roodman) allows you to estimate using MLE a wide range of simultaneous models with combinations of linear and non-linear equations provide they satisfy a recursive structure.

treatreg allows the estimation of what Stata calls "treatment effects models". This is something of a misnomer since it only for a very specific model: a linear regression with an endogenous dummy.

condivreg estimates IV models with a single endogenous variable and provides an exact confidence interval for the slope as opposed to the usual asymptotic one. It is particularly useful if weak instruments are a concern.

For estimating Treatment effects using Propensity Score matching there are several downloads including: psmatch2 (Leuven & Sianesi) which does a wide range of matching estimators and nnmatch which does nearest neighbour matching. psbalance allows you to test covariate balance after matching - something that is recomended.

Matching Methods for Causal Inference: A Review and a Look Forward

Author: Elizabeth A. Stuart

Source: Statist. Sci. Volume 25, Number 1 (2010), 1-21

Abstract:

When estimating causal effects using observational data, it is desirable to replicate a randomized experiment as closely as possible by obtaining treated and control groups with similar covariate distributions. This goal can often be achieved by choosing well-matched samples of the original treated and control groups, thereby reducing bias due to the covariates. Since the 1970s, work on matching methods has examined how to best choose treated and control subjects for comparison. Matching methods are gaining popularity in fields such as economics, epidemiology, medicine and political science. However, until now the literature and related advice has been scattered across disciplines. Researchers who are interested in using matching methods—or developing methods related to matching—do not have a single place to turn to learn about past and current research. This paper provides a structure for thinking about matching methods and guidance on their use, coalescing the existing research (both old and new) and providing a summary of where the literature on matching methods is now and where it should be headed.

Keywords: Observational study; propensity scores; subclassification; weighting

Evaluating State Programmes: “Natural Experiments” and Propensity Scores

DENIS CONNIFFE, VANESSA GASH, PHILIP J. O’CONNELL

The Economic and Social Review, Vol. 31, No. 4, October, 2000, pp. 283-308

Abstract: Evaluations of programmes — for example, labour market interventions such as employment schemes and training courses — usually involve comparison of the performance of a treatment group (recipients of the programme) with a control group (non-recipients) as regards some response (gaining employment, for example). But the ideal of randomisation of individuals to groups is rarely possible in the social sciences and there may be substantial differences between groups in the distributions of individual characteristics that can affect response. Past practice in economics has been to try to use multiple regression models to adjust away the differences in observed characteristics, while also testing for sample selection bias. The Propensity Score approach, which is widely applied in epidemiology and related fields, focuses on the idea that “matching” individuals in the groups should be compared. The appropriate matching measure is usually taken to be the prior probability of programme participation. This paper describes the key ideas of the Propensity Score method and illustrates its application by reanalysis of some Irish data on training courses.

An Article on Job Re-Training in Yesterday's NYT

"In Michigan, where the unemployment rate in May was 14.1 percent, the nation’s highest, 78,000 people are enrolled in the state’s No Worker Left Behind program and 7,800 are on the waiting list...

...a little-noticed study the Labor Department released several months ago found that the benefits of the biggest federal job training program (the Workforce Investment Act) were “small or nonexistent” for laid-off workers...

...economists cited several reasons (for why) retraining might not be effective. Many workers who have lost their jobs are older and had spent their lives working in one industry. In need of a job right away, many pick relatively short training programs, which often have marginal benefits. Job retraining is also ineffective without job creation..."

The full article from the NYT is available here. Deatils of Michigan's "No Worker Left Behind" program are available here. Information about the Workforce Investment Act is available here. The study mentioned in the NYT article ("Workforce Investment Act (WIA) Non-Experimental Net Impact Evaluation") was conducted by Impaq International and is available here since December 2008.

Propensity score matching is used in the study to identify individuals in comparison groups who are similar to the individuals who participated in the WIA program. The study uses administrative data from 12 states, dividing the data for each state into three classes: base data, comprising WIA program participants; comparison data, providing information on individuals in other programs who are matched to treated cases; and outcome data, merged by individual identifier. The states in the study are Connecticut, Indiana, Kentucky, Maryland, Missouri, Minnesota, Mississippi, Montana, New Mexico, Tennessee, Utah, and Wisconsin.