Evaluating State Programmes: “Natural Experiments” and Propensity Scores

DENIS CONNIFFE, VANESSA GASH, PHILIP J. O’CONNELL

The Economic and Social Review, Vol. 31, No. 4, October, 2000, pp. 283-308

Abstract: Evaluations of programmes — for example, labour market interventions such as employment schemes and training courses — usually involve comparison of the performance of a treatment group (recipients of the programme) with a control group (non-recipients) as regards some response (gaining employment, for example). But the ideal of randomisation of individuals to groups is rarely possible in the social sciences and there may be substantial differences between groups in the distributions of individual characteristics that can affect response. Past practice in economics has been to try to use multiple regression models to adjust away the differences in observed characteristics, while also testing for sample selection bias. The Propensity Score approach, which is widely applied in epidemiology and related fields, focuses on the idea that “matching” individuals in the groups should be compared. The appropriate matching measure is usually taken to be the prior probability of programme participation. This paper describes the key ideas of the Propensity Score method and illustrates its application by reanalysis of some Irish data on training courses.

Mostly Harmless!

DON'T PANIC! The core methods in today's econometric toolkit are linear regression for statistical control, instrumental variables methods for the analysis of natural experiments, and differences-in-differences methods that exploit policy changes.

This is the refrain of Joshua Angrist and Steve Pischke in the preface to their new book: "Mostly Harmless Econometrics". We mentioned it on the blog before here. In advance of Professor Angrist's visit to Geary on Friday, people might be interested in a preview of the Mostly Harmless book that is available here. For anyone who doesn't have a copy of the book (or even if you do), it's worth looking at the paper by Angrist and Krueger on "Empirical Strategies in Labour Economics". This is more a primer than a paper; it featured here (takes less time to to load up) in the 1999 Handbook of Labour Economics.

Abstract below:

This chapter provides an overview of the methodological and practical issues that arise when estimating causal relationships that are of interest to labor economists. The subject matter includes identification, data collection, and measurement problems. Four identification strategies are discussed, and five empirical examples -- the effects of schooling, unions, immigration, military service, and class size -- illustrate the methodological points. In discussing each example, we adopt an experimentalist perspective that emphasizes the distinction between variables that have causal effects, control variables, and outcome variables. The chapter also discusses secondary datasets, primary data collection strategies, and administrative data. The section on measurement issues focuses on recent empirical examples, presents a summary of empirical findings on the reliability of key labor market data, and briefly reviews the role of survey sampling weights and the allocation of missing values in empirical research.

Finally, there is also a range of "Mostly Harmless" t-shirts for sale; see below for a sample!

Advertising Works - According to Yahoo! Research

We have discussed the possibility of evaluating adevertising campaigns on this blog before (here and here). So it is interesting to read that Yahoo! Research is measuring the effects of advertising on sales through a controlled experiment (more details available here). Researcher David Reiley has recently collaborated with Yahoo!’s Marketing Insights team in their ongoing efforts to help advertisers evaluate the effectiveness of their advertising campaigns.

Reiley joined Yahoo! Research because of his longstanding interest in field experiments and he points out a potential weakness of a study reported in a recent Harvard Business Review article, which measured large increases in sales due to online advertising. The study used large quantities of data from comScore, a key online information provider that logs the Internet browsing behaviour of two million users worldwide. By comparing the purchases of those who saw a given online ad with the purchases of those who do did not see it, the study concluded that there are large positive effects of online advertising.

However, "the population of people who sees a particular ad may be very different from the population who does not see the same ad,” says Reiley. Reiley's research made use of a database match between Yahoo! and a nationwide retailer by identifying users who registered the same email address with both companies. After finding over one million matched users, the researchers randomly assigned them to treatment and control groups for one of the retailer’s online advertising campaigns.

The project then tracked sales each week at the retailer, both online and in stores. In a paper co-authored with summer intern and MIT PhD student Randall Lewis, Reiley found that the online display advertising increased total revenues by approximately 5% for those users exposed to the ads, with 93% of the total effect happening in offline sales. They also observed online ads to have a large impact on sales even when the ads are not clicked: 78% of the increase in sales came from those who viewed, but did not click, the ads.

Could this have implications for the "pay-per-click" model in online advertising?

Mostly Harmless Econometrics

DON'T PANIC! The core methods in today's econometric toolkit are linear regression for statistical control, instrumental variables methods for the analysis of natural experiments, and differences-in-differences methods that exploit policy changes.

This is the refrain of Joshua Angrist and Steve Pischke in the preface to their new book: "Mostly Harmless Econometrics". A preview is available here. My econometrics professor, Paul Devereux, tells me that there are many references throughout to the Hitchhikers Guide to the Galaxy. There is also a range of "Mostly Harmless" t-shirts for sale; see below for a sample!