A Lesson in Policy Evaluation: Moving to Opportunity

“Moving to Opportunity” (MTO) is a very fine example of a well thought-out evaluation of a potential policy aimed at tackling one (possibly) important source of disadvantage for low income families. There has long been a debate about the role of neighbourhood effects in determining outcomes; does living in a high poverty environment adversely affect your life chances, would individuals in these neighbourhoods be badly off anyway where ever they were, or do they live there because they are have bad economic outcomes? Any decent social scientist (or indeed anyone interested in evidenced based policies) will tell you that these explanations are extremely difficult to disentangle.

The MTO RCT involved providing families (4,600 of them) in public housing with vouchers to move to less disadvantaged neighbourhoods in 5 US cities (Baltimore, Boston, Chicago, LA and New York) in the mid 1990s. Conducted in association with the Department of Housing and Urban Development, the programme has been rigorously evaluated by a team of researchers including Jens Ludwig of Chicago and Larry Katz of Harvard. It goes without saying, or perhaps it doesn’t in Ireland, that the use of a lottery for voucher allocation is what makes this design so powerful. The final impact evaluation is currently being completed; previous academic papers are available below.

http://www.economics.harvard.edu/faculty/katz/papers_katz

A summary of the findings is available here.

http://www.huduser.org/publications/pdf/MTOFHD_summaryreport.pdf

Essentially there was (perhaps surprisingly) little effect of moving to a better neighbhourhood on the economic outcomes of families, or the test scores of children. On the other hand, there were important differences in terms of both physical (in particular obesity and diabetes) and mental health between experimental and control groups. The experimental group also reported higher levels of life satisfaction. On this basis Larry Katz summarized the effects of the treatment as being more a case of “Moving to Tranquility” than “Moving to Opportunity”.

One other important aspect of this programme is a willingness on behalf of the organizers to engage with other researchers, provide data and debate the findings. This adds greatly to the credibility of these studies, something else which does not seem to have been grasped in Ireland. Making data publicly available also goes some way to alleviating concerns about the influence of those commissioning the research. For an example of this, see the following exchange in the American Journal of Sociology on MTO.

Clampet-Lundquist, Susan, and Douglas S. Massey. 2008. “Neighborhood Effects on Economic Self-Sufficiency: A Reconsideration of the Moving to Opportunity Experiment.” American Journal of Sociology 114 (1): 107–43.

http://www.bsos.umd.edu/socy/vanneman/socy789b/ClampetLundquistM08.pdf

Abstract:

This article revisits the Moving to Opportunity housing mobility experiment, which heretofore has not provided strong evidence to support the hypothesis of neighborhood effects on economic self-sufficiency among adults. The authors undertake a conceptual and empirical analysis of the study’s design and implementation to gain a better understanding of the selection processes that occur within the study. The article shows that the study is potentially affected by selectivity at several junctures: in determining who complied with the program’s requirements, who entered integrated versus segregated neighborhoods, and who left neighborhoods after initial relocation. Furthermore, previous researchers have not found an experimental treatment effect on adult economic self-sufficiency, relative to controls. The authors propose an alternative approach that involves measuring the cumulative amount of time spent in different neighborhood environments. With this method, they find evidence that neighborhood is associated with outcomes such as employment, earnings, TANF receipt, and use of food stamps.

What Can We Learn about Neighborhood Effects from the Moving to Opportunity Experiment? Jens Ludwig, Jeffrey B. Liebman, Jeffrey R. Kling, Greg J. Duncan, Lawrence F. Katz, Ronald C. Kessler, and Lisa Sanbonmatsu, American Journal of Sociology 114 (July 2008), 144-88.


http://www.economics.harvard.edu/faculty/katz/files/What%20Can%20We%20Learn%20about%20Neighborhood%20Effects.pdf

Abstract:

Experimental estimates from Moving to Opportunity (MTO) show no significant impacts of moves to lower-poverty neighborhoods on adult economic self-sufficiency four to seven years after random assignment. The authors disagree with Clampet-Lundquist and Massey’s claim that MTO was a weak intervention and therefore uninformative about neighborhood effects. MTO produced large changes in neighborhood environments that improved adult mental health and many outcomes for young females. Clampet-Lundquist and Massey’s claim that MTO experimental estimates are plagued by selection bias is erroneous. Their new nonexperimental estimates are uninformative because they add back the selection problems that MTO’s experimental design was intended to overcome.

Who is in charge of the economy?

I think this quote from Paul Seabright's The company of strangers: a natural history of economic life is well worth thinking about:
"Politicians are in charge of the modern economy in much the same way as a sailor is in charge of a small boat in a storm. The consequences of their losing control completely may be catastrophic (as civil war and hyperinflation in parts of the former Soviet empire have recently reminded us), but even while they keep afloat, their influence over the course of events is tiny in comparison with that of the storm around them. We who are their passengers may focus our hopes and fears upon them, and express profound gratitude toward them if we reach harbor safely, but that is chiefly because it seems pointless to thank the storm. (p. 25)"

(via Bill Easterly via Greg Mankiw via Peter Gordon)

Links: 23rd July. Special Focus: Emigration.

1. Adam Jaffe: Building Program Evaluation Into The Design Of Public Research Support Programs

2. NSF Survey of Federal Funds for Research and Development

3. Greg Mankiw discusses the U.S. Council of Economic Advisers' "impossible job": measuring how many jobs were created by the American Recovery and Reinvestment Act of 2009. Here's a link to the CEA document.

4. Marginal Revolution flags a recent paper by Weber and Castillo from Yahoo! Research: The Demographics of Web Search. "How does the web search behavior of 'rich' and 'poor' people differ? Do men and women tend to click on different results for the same query?"

5. David McWilliams on emigration out of Ireland: "The ESRI predicts that 120,000 will leave the country in the next 18 months, on top of the 100,000 who have already gone in the past 18 months."

6. It was last year that the CSO reported that for the first time since 1995 more people left Ireland than moved here. That is, there was net outward migration.

7. Overall, the ESRI commentary warns that 200,000 people may be forced to emigrate between now and 2015 if unemployment does not abate.

8. In addition, a new Eurobarometer survey shows more than one fifth of Irish people are thinking of moving abroad for work. It also shows that 35% of Irish people feel employment prospects are better in other countries.

9. This Irish Times article states that current statistics show "Irish people are moving to both Canada and Australia in increasing numbers to escape the economic recession but are largely shunning traditional emigration routes to Britain and the US."

10. A companion article by the same journalist (Jamie Smyth) considers whether time abroad could end up as permanent emigration: "...if the Irish economy does not rebound in the next few years and create job opportunities for the next generation of graduates and school-leavers it seems very likely that a year spent working abroad will lead to longer term and more permanent emigration." To quote an old Pogues song, (hundreds of) thousands are "sailing".

An evaluation of a re-offender program

Pro Bono Economics "is a new charity whose aim is to broker economists into the charitable sector to help on short and medium-term assignments, typically addressing questions around measurement, results and impact.
Charitable work can have wider social and economic benefits. Quantitative evaluations by economists can help in measuring those benefits and in determining the most effective ways of channelling support to achieve them."
They have just produced their first report, which was provided by Frontier Economics an excellent economics consultancy based in London. The report "evaluates the benefits of the St Giles’ “Through the Gates” programme. The programme is aimed at reducing the rate of re-offending. It provides a range of services including pre-release assessment and, following release, accommodation and help in reintegration and employment search. In examining the economic impact of the Through the Gates programme at St Giles, Frontier Economics has confirmed the value of this important work."
Frontier estimates Benefit/Cost ratios of the order of 10 to 1. This report is a really good read, a good example of economics applied to an important topic and of the Pro Bono project.

http://www.probonoeconomics.com/pdf/St_Giles.pdf
(h/t Colm Harmon)

http://www.frontier-economics.com/

Some Links

1. It's not your peers, and it's not your friends: Some progress toward understanding the educational peer effect mechanism

2. Econometric Methods for Causal Evaluation of Education Policies and Practices:
A Non-Technical Guide

3. Denis the Dentist: people disproportionately choose careers whose labels resemble their names

4. http://www.babynamewizard.com/

5. Penelope Trunk: Brazen Careerist

6. Randomness, Luck, and other Situational Sources of Success and Failure

7. Why jobs obtained through networks may lower your earnings

8. Vive la Révolution!: Long term returns of 1968 to the angry students

9. Simultaneous behaviour: during the Winter Olympics in Edmonton (Canada) (HT: Eoin Rouine)

Doctor, doctor! How's business?

Could always be better.

I don't find the Dr., dr., jokes very funny either, but their traditional popularity is difficult to contend with..

Here's the latest one :

What does this policy achieve?

(besides insulting everyone with the slightest intelligence)

The poorest 35% of the population already visit the GP for free in Ireland. The remaining 65% are able to claim tax relief on all their GP expenses.

What is behind this idea? I'm not aware of any evidence to suggest that it might improve health. In fact there is much more evidence to suggest that it will exacerbate problems in health care; for one, it amounts to "over-insurance" which is one of the main drivers of excessive medical inflation. I do expect there to be something more to this plan than the slogan, but at any rate, it seems like more of the unfortunate "here you go, vote for me"style of politics that we know served us poorly in the past. By applying the strategy to healthcare they are totally undermining their political credibility. Why don't they just be honest about it and hand out fifties - it would be cheaper and wouldn't lengthen queues and shorten examination times at the GP's clinic.

One obvious place to start is with the recommendations of the recent Competition Authority Report about GP practices.

Randomised Evaluation of a Parenting Programme

Donal O'Neill presented at the economics seminar on Friday. He has a new working paper on an early childhood intervention in Ireland.



A Cost-Benefit Analysis of Early Childhood Intervention: Evidence from a Randomised Evaluation of a Parenting Programme
by Donal O'Neill (October 2009)

Abstract:
A number of researchers and policy makers have recently argued that the most effective way of dealing with long-run disadvantage and the intergenerational transmission of poverty is through early childhood intervention and in particular policies aimed at supporting the family in early childhood development. In this paper we carry out a randomised evaluation of one such programme aimed at improving the skills and parenting strategies of parents, particularly those who find their child's behaviour difficult or challenging. Our evaluation shows that the treatment significantly reduced behavioural problems in young children when measured 6 months after the intervention. Furthermore our detailed cost analysis, combined with a consideration of the potential long-run benefits associated with the programme, suggest that the long-run rate of return to society from this programme is likely to be relatively high.

Evaluating State Programmes: “Natural Experiments” and Propensity Scores

DENIS CONNIFFE, VANESSA GASH, PHILIP J. O’CONNELL

The Economic and Social Review, Vol. 31, No. 4, October, 2000, pp. 283-308

Abstract: Evaluations of programmes — for example, labour market interventions such as employment schemes and training courses — usually involve comparison of the performance of a treatment group (recipients of the programme) with a control group (non-recipients) as regards some response (gaining employment, for example). But the ideal of randomisation of individuals to groups is rarely possible in the social sciences and there may be substantial differences between groups in the distributions of individual characteristics that can affect response. Past practice in economics has been to try to use multiple regression models to adjust away the differences in observed characteristics, while also testing for sample selection bias. The Propensity Score approach, which is widely applied in epidemiology and related fields, focuses on the idea that “matching” individuals in the groups should be compared. The appropriate matching measure is usually taken to be the prior probability of programme participation. This paper describes the key ideas of the Propensity Score method and illustrates its application by reanalysis of some Irish data on training courses.

Policy Design and Evaluation

Very little of the debate surrounding the McCarthy report has focused on actual economic evidence, and the report itself does not contain any references to scientific articles in Economics.

This is not necessarily a criticism of the report. But it should be remembered that the name "Bord Snip" is a joke name and not the title of the report, which is actually Report of the "Special Group on Public Service Numbers and Expenditure Programmes".

Given the title of such a document, people who lived in a cave and read only this blog and surfaced the odd time for seminars might have expected intense discussion about the precise effect of class sizes on student attainment, or references to randomised trials examining the cost-effectiveness of different types of social welfare systems.

What the report delivered was solid, old-style, common-sense suggestions for where budgets in different departmental divisions had become too fat along with solid recommendations for reducing this. The report is written in very plain English and the authors are in no sense open to allegations of fudging their task or being vague.

It does leave open questions on what should be the status of modern economic analysis in the future shape of public sector development in Ireland. The report leans heavily toward the implementation of traditional solid CBA analysis linked to performance reviews, laudable in themselves but certainly something that could be done without any knowledge of recent literatures in econometrics, behavioural economics, programme design and so on.

The question people who are working in these fields need to ask is whether the work going on in these areas should be having more influence on the discussions on policy in Ireland in areas such as pensions, taxation, aging, health, education, social welfare, transport and so on. I have been fortunate enough to debate these issues with many people in business and policy and views range from a strong belief that policy design is being crippled by a failure to engage properly with new thinking in economics to a view that anything beyond solid accounting and strategic management of targeted policies is a waste of money.

Obviously, my own view leans toward a belief that the new literatures in economics are teaching us a lot about the policy issues that an aging population in Ireland and Europe will face. The bring together of economics, psychology, public health and related fields linked to people who have expertise in the implementation and oversite of policy has, in my view, the potential to dramatically improve the way policies are developed and delivered. At present, such literatures are at the very margins of conciousness for most of the people charged with developing and implementing policy. Economics became a very strong force in countries like Ireland as acceptance emerged of principles of regulation and market systems emanating from the academic literature. The current academic literature is increasingly dominated by accounts of agents with imperfect information and bounded rationality but, as yet, with very small policy impact even internationally (though clearly Thaler and Sunnstein are becoming increasingly discussed and the latter now has a policy position). For people who read this blog, the question is whether behavioural economics is a meaningful career path outside of academia. Will the policymakers, evaluators, business leaders and so on in the next few years be as influenced by the current literature as the older generation were by their literature? A lot will depend on how the US literature and policy landscape develops but a lot more thinking is also needed here.

Alternative Methods for Evaluation - Costas-Dias and Blundell

Thanks to Christian for pointing out that an updated version of this excellent review is available online.

http://www.fep.up.pt/investigacao/cete/papers/DP0805.pdf