New York Times Article on Randomisation Designs for Housing Policy

An interesting article in the NYT covers a debate about the use of an RCT design to test the efficacy of homelessness prevention programme called HOMEBASE. Opponents of the design argue that it wasn't need given that the programme was already demonstrating efficacy and also that it is unethical to treat people like lab rats in this fashion. Proponents argue that the money is currently being spent without awareness of whether it is really having an effect and that, therefore, the use of the RCT design will be a powerful guide as to whether the programme should be continued and/or scaled up.

Secrets and lies: is a lot of medical research bunk?

Social scientists & certainly economists tend to look up to medical research, partly because its where the money is, and also because one of its key methods, the randomized control trial, is seen by many as providing a "gold standard" when it comes to measuring treatment effects - though Deaton, Heckman and others have questioned whether RCTs in economics should enjoy this privileged status.
The public generally tend to hold medical research in even higher respect. Medical researchers are good people, passionately if objectively, pushing back the frontiers of knowledge to help make us better.
So how worrying would it be if much medical research was actually wrong? This conclusion has been emerging from the work of a Greek medical researcher, John Ioannidis, and his team. The causes of this problem are various, including publication bias, and are well known but the scale of the problem is probably not. This article may make you distinctly uneasy.
Part of the problem, that key studies have not been replicated and may be wrong, is not peculiar to medicine. It may well plague the social and behavioural sciences too. This article on the subject won't make you feel any better.

Cartwright on the value of Randomised Trials

Thanks to Graham Brownlow on irisheconomy blog for pointing to this gem (requires subscription).

Nancy Cartwright, ‘What are randomised controlled trials good for? ’, Philosophical Studies, vol. 147, no.1 (2010), pp.59-70.

Abstract Randomized controlled trials (RCTs) are widely taken as the gold standard for establishing causal conclusions. Ideally conducted they ensure that the treatment ‘causes’ the outcome—in the experiment. But where else? This is the venerable question of external validity. I point out that the question comes in two importantly different forms: Is the specific causal conclusion warranted by the experiment true in a target situation? What will be the result of implementing the treatment there? This paper explains how the probabilistic theory of causality implies that RCTs can establish causal conclusions and thereby provides an account of what exactly that causal conclusion is. Clarifying the exact form of the conclusion shows just what is necessary for it to hold in a new setting and also how much more is needed to see what the actual outcome would be there were the treatment implemented.

Keywords Randomized controlled trials (RCTs) External validity
Probabilistic theory of causality Causal inference Capacities Contributions

Randomised Trials and Development

Easterly outlines a new conference book on the topic and gives a good feel for the debate

link here 

Ethics and Economics of Randomised Trials

good discussion on Tyler Cowen's blog on this

link here

NBER Videos on the Use of Field Experiments in Economics

John List and Michael Kremer talk about the design and use of experiments in economics

link here

Mostly Harmless!

DON'T PANIC! The core methods in today's econometric toolkit are linear regression for statistical control, instrumental variables methods for the analysis of natural experiments, and differences-in-differences methods that exploit policy changes.

This is the refrain of Joshua Angrist and Steve Pischke in the preface to their new book: "Mostly Harmless Econometrics". We mentioned it on the blog before here. In advance of Professor Angrist's visit to Geary on Friday, people might be interested in a preview of the Mostly Harmless book that is available here. For anyone who doesn't have a copy of the book (or even if you do), it's worth looking at the paper by Angrist and Krueger on "Empirical Strategies in Labour Economics". This is more a primer than a paper; it featured here (takes less time to to load up) in the 1999 Handbook of Labour Economics.

Abstract below:

This chapter provides an overview of the methodological and practical issues that arise when estimating causal relationships that are of interest to labor economists. The subject matter includes identification, data collection, and measurement problems. Four identification strategies are discussed, and five empirical examples -- the effects of schooling, unions, immigration, military service, and class size -- illustrate the methodological points. In discussing each example, we adopt an experimentalist perspective that emphasizes the distinction between variables that have causal effects, control variables, and outcome variables. The chapter also discusses secondary datasets, primary data collection strategies, and administrative data. The section on measurement issues focuses on recent empirical examples, presents a summary of empirical findings on the reliability of key labor market data, and briefly reviews the role of survey sampling weights and the allocation of missing values in empirical research.

Finally, there is also a range of "Mostly Harmless" t-shirts for sale; see below for a sample!

Reevaluating Learning - NBER Summary Duflo

From the NBER site, an examination of factors that improve student outcomes from Esther Duflo. The review is based on experimental evaluations of changes in remedial education, classroom organisation and teacher incentives, among others

http://www.nber.org/reporter/2009number1/duflo.html

Could Advertising Be Part of the Recovery?

As I've noticed shorter and shorter ad breaks on TV and radio over the last month, I've been wondering if lower advertising revenues are having an effect on jobs in the media sector. In some sense I'm glad to have less interruptions during my media consumption, but media, multimedia, graphic design and advertising are popular career choices in Ireland, and it doesn't appeal to think of employment opportunities in this sector of the economy drying up.

Before Christmas, a story emerged that cuts might be made at Irish televisions stations due to lower advertising revenues. After Christmas we heard that Irish advertising spend was down 30 per cent at end of 2008, and is predicted to fall further in 2009. On the other side of the Atlantic, a pay freeze was announced yesterday at Yahoo.

The question arises - could more jobs in media, multimedia, graphic design and advertising be saved if advertising revenues didn't fall so much? Also, is there a commercial imperative for companies (who are in it for the long haul) to keep advertising during the recession (and thereby stimulate employment in some parts of the economy)?

In other words, could advertising be part of the recovery? We mentioned recently that the Economist journal review points to an interesting article from Knowledge@Wharton: "When the going gets tough, the tough don't skimp on their ad budgets". According to the article:

"Research shows that companies that consistently advertise even during recessions perform better in the long run. A McGraw-Hill Research study looking at 600 companies from 1980 to 1985 found that those businesses which chose to maintain or raise their level of advertising expenditures during the 1981 and 1982 recession had significantly higher sales after the economy recovered. Specifically, companies that advertised aggressively during the recession had sales 256% higher than those that did not continue to advertise."

It may not be as simple as simply pouring money into advertising campaigns though. First of all, if cash is scarce, how can campaigns be financed? Maybe a special bond could be set up by IBEC and/or ISME and issued to the Irish public?

Second, while there is evidence that advertising during a recession is beneficial, the extent (or even the existence) of the benefit may depend on the nature of business being conducted. Some very brief google-searching on the topic brings up the Ad Contrarian Blog (run by the CEO of an ad agency). This post on advertsting during a recession, suggests that there are three types of businesses (in relation to advertising), as follows:

* Businesses for whom advertising is essential.
* Businesses for whom advertising is discretionary.
* Businesses that are somewhere in the middle.

According to the post, businesses for whom advertising is essential tend to be in categories that are consumer-oriented and highly responsive to marketing activity. They include fast food, retail, consumer packaged goods, and automotive. This suggests that Irish companies involved in these types of activities should ensure that they do not cut back on advertising during the recession.

Also, according to this post on the same blog, it is better to focus on changing behavior rather then attitudes. Apparently, it is easier to convince someone to eat a Big Mac than to convince them that a Big Mac is a good thing to eat. Also, it is easier to convince someone to go to Las Vegas than to convince them that going to Las Vegas is a smart thing to do.

The question remains though, how can companies test that particular advertising approaches during a recession will be (more or less) successful? We mentioned before (here) that Google has rolled out a tool designed to show how people browsing the web respond to online advertising (Adplanner - see here).

"By comparing the behaviour of people who are exposed to particular adverts with those who are not, the company hopes to give advertisers feedback about which campaigns are are successful." This may be one of the first instances where advertising has been evaluated in the spirit of randomised controlled trials.

Mostly Harmless Econometrics

DON'T PANIC! The core methods in today's econometric toolkit are linear regression for statistical control, instrumental variables methods for the analysis of natural experiments, and differences-in-differences methods that exploit policy changes.

This is the refrain of Joshua Angrist and Steve Pischke in the preface to their new book: "Mostly Harmless Econometrics". A preview is available here. My econometrics professor, Paul Devereux, tells me that there are many references throughout to the Hitchhikers Guide to the Galaxy. There is also a range of "Mostly Harmless" t-shirts for sale; see below for a sample!