A Lesson in Policy Evaluation: Moving to Opportunity

“Moving to Opportunity” (MTO) is a very fine example of a well thought-out evaluation of a potential policy aimed at tackling one (possibly) important source of disadvantage for low income families. There has long been a debate about the role of neighbourhood effects in determining outcomes; does living in a high poverty environment adversely affect your life chances, would individuals in these neighbourhoods be badly off anyway where ever they were, or do they live there because they are have bad economic outcomes? Any decent social scientist (or indeed anyone interested in evidenced based policies) will tell you that these explanations are extremely difficult to disentangle.

The MTO RCT involved providing families (4,600 of them) in public housing with vouchers to move to less disadvantaged neighbourhoods in 5 US cities (Baltimore, Boston, Chicago, LA and New York) in the mid 1990s. Conducted in association with the Department of Housing and Urban Development, the programme has been rigorously evaluated by a team of researchers including Jens Ludwig of Chicago and Larry Katz of Harvard. It goes without saying, or perhaps it doesn’t in Ireland, that the use of a lottery for voucher allocation is what makes this design so powerful. The final impact evaluation is currently being completed; previous academic papers are available below.

http://www.economics.harvard.edu/faculty/katz/papers_katz

A summary of the findings is available here.

http://www.huduser.org/publications/pdf/MTOFHD_summaryreport.pdf

Essentially there was (perhaps surprisingly) little effect of moving to a better neighbhourhood on the economic outcomes of families, or the test scores of children. On the other hand, there were important differences in terms of both physical (in particular obesity and diabetes) and mental health between experimental and control groups. The experimental group also reported higher levels of life satisfaction. On this basis Larry Katz summarized the effects of the treatment as being more a case of “Moving to Tranquility” than “Moving to Opportunity”.

One other important aspect of this programme is a willingness on behalf of the organizers to engage with other researchers, provide data and debate the findings. This adds greatly to the credibility of these studies, something else which does not seem to have been grasped in Ireland. Making data publicly available also goes some way to alleviating concerns about the influence of those commissioning the research. For an example of this, see the following exchange in the American Journal of Sociology on MTO.

Clampet-Lundquist, Susan, and Douglas S. Massey. 2008. “Neighborhood Effects on Economic Self-Sufficiency: A Reconsideration of the Moving to Opportunity Experiment.” American Journal of Sociology 114 (1): 107–43.

http://www.bsos.umd.edu/socy/vanneman/socy789b/ClampetLundquistM08.pdf

Abstract:

This article revisits the Moving to Opportunity housing mobility experiment, which heretofore has not provided strong evidence to support the hypothesis of neighborhood effects on economic self-sufficiency among adults. The authors undertake a conceptual and empirical analysis of the study’s design and implementation to gain a better understanding of the selection processes that occur within the study. The article shows that the study is potentially affected by selectivity at several junctures: in determining who complied with the program’s requirements, who entered integrated versus segregated neighborhoods, and who left neighborhoods after initial relocation. Furthermore, previous researchers have not found an experimental treatment effect on adult economic self-sufficiency, relative to controls. The authors propose an alternative approach that involves measuring the cumulative amount of time spent in different neighborhood environments. With this method, they find evidence that neighborhood is associated with outcomes such as employment, earnings, TANF receipt, and use of food stamps.

What Can We Learn about Neighborhood Effects from the Moving to Opportunity Experiment? Jens Ludwig, Jeffrey B. Liebman, Jeffrey R. Kling, Greg J. Duncan, Lawrence F. Katz, Ronald C. Kessler, and Lisa Sanbonmatsu, American Journal of Sociology 114 (July 2008), 144-88.


http://www.economics.harvard.edu/faculty/katz/files/What%20Can%20We%20Learn%20about%20Neighborhood%20Effects.pdf

Abstract:

Experimental estimates from Moving to Opportunity (MTO) show no significant impacts of moves to lower-poverty neighborhoods on adult economic self-sufficiency four to seven years after random assignment. The authors disagree with Clampet-Lundquist and Massey’s claim that MTO was a weak intervention and therefore uninformative about neighborhood effects. MTO produced large changes in neighborhood environments that improved adult mental health and many outcomes for young females. Clampet-Lundquist and Massey’s claim that MTO experimental estimates are plagued by selection bias is erroneous. Their new nonexperimental estimates are uninformative because they add back the selection problems that MTO’s experimental design was intended to overcome.

5000 pounds offered to students who fail A-levels

One school in the UK is so confident of its success rate that it is offering 5k to any student who doesn't pass their A-levels, conditional on them having good attendance and assignment submission records. The BBC reports "It is the latest example of cash or gift incentives being used in schools - either to encourage good behaviour or to discourage bad." At first glance, offering teenagers money to fail seems counterintuitive as an incentive for performance. But as a signal of the school's confidence in its quality, it clearly has some advantages. However, they should be careful. If I were the type of teenager who valued 5k over my education or if I was pretty sure I was going to fail anyway, then this school would look pretty appealing! Also, marginal teenagers might think that 5k is a lot of money and might also be discounting the future at a very high rate.

Mindspace UK

Ever wondered what drives us to behave in the way we do? from Matrix Knowledge Group on Vimeo.

Doctor, doctor! How's business?

Could always be better.

I don't find the Dr., dr., jokes very funny either, but their traditional popularity is difficult to contend with..

Here's the latest one :

What does this policy achieve?

(besides insulting everyone with the slightest intelligence)

The poorest 35% of the population already visit the GP for free in Ireland. The remaining 65% are able to claim tax relief on all their GP expenses.

What is behind this idea? I'm not aware of any evidence to suggest that it might improve health. In fact there is much more evidence to suggest that it will exacerbate problems in health care; for one, it amounts to "over-insurance" which is one of the main drivers of excessive medical inflation. I do expect there to be something more to this plan than the slogan, but at any rate, it seems like more of the unfortunate "here you go, vote for me"style of politics that we know served us poorly in the past. By applying the strategy to healthcare they are totally undermining their political credibility. Why don't they just be honest about it and hand out fifties - it would be cheaper and wouldn't lengthen queues and shorten examination times at the GP's clinic.

One obvious place to start is with the recommendations of the recent Competition Authority Report about GP practices.

Policy Design and Evaluation

Very little of the debate surrounding the McCarthy report has focused on actual economic evidence, and the report itself does not contain any references to scientific articles in Economics.

This is not necessarily a criticism of the report. But it should be remembered that the name "Bord Snip" is a joke name and not the title of the report, which is actually Report of the "Special Group on Public Service Numbers and Expenditure Programmes".

Given the title of such a document, people who lived in a cave and read only this blog and surfaced the odd time for seminars might have expected intense discussion about the precise effect of class sizes on student attainment, or references to randomised trials examining the cost-effectiveness of different types of social welfare systems.

What the report delivered was solid, old-style, common-sense suggestions for where budgets in different departmental divisions had become too fat along with solid recommendations for reducing this. The report is written in very plain English and the authors are in no sense open to allegations of fudging their task or being vague.

It does leave open questions on what should be the status of modern economic analysis in the future shape of public sector development in Ireland. The report leans heavily toward the implementation of traditional solid CBA analysis linked to performance reviews, laudable in themselves but certainly something that could be done without any knowledge of recent literatures in econometrics, behavioural economics, programme design and so on.

The question people who are working in these fields need to ask is whether the work going on in these areas should be having more influence on the discussions on policy in Ireland in areas such as pensions, taxation, aging, health, education, social welfare, transport and so on. I have been fortunate enough to debate these issues with many people in business and policy and views range from a strong belief that policy design is being crippled by a failure to engage properly with new thinking in economics to a view that anything beyond solid accounting and strategic management of targeted policies is a waste of money.

Obviously, my own view leans toward a belief that the new literatures in economics are teaching us a lot about the policy issues that an aging population in Ireland and Europe will face. The bring together of economics, psychology, public health and related fields linked to people who have expertise in the implementation and oversite of policy has, in my view, the potential to dramatically improve the way policies are developed and delivered. At present, such literatures are at the very margins of conciousness for most of the people charged with developing and implementing policy. Economics became a very strong force in countries like Ireland as acceptance emerged of principles of regulation and market systems emanating from the academic literature. The current academic literature is increasingly dominated by accounts of agents with imperfect information and bounded rationality but, as yet, with very small policy impact even internationally (though clearly Thaler and Sunnstein are becoming increasingly discussed and the latter now has a policy position). For people who read this blog, the question is whether behavioural economics is a meaningful career path outside of academia. Will the policymakers, evaluators, business leaders and so on in the next few years be as influenced by the current literature as the older generation were by their literature? A lot will depend on how the US literature and policy landscape develops but a lot more thinking is also needed here.

Principles of Policy Design and Evaluation

Below is the relevant section from the McCarthy Report on principles that should guide policy design and evaluation in Ireland. It is important that this blog discusses the extent to which the econometric and behavioural literature in this field can actually improve public policy. The work of Heckman on the evaluation of public programmes and the nature of life-cycle inequality is particularly important in my view.

From Chapter 2 of Volume 1.

(i) Every proposed new spending programme should be accompanied by a Public Service Performance Charter, which sets out clearly the business case for the programme, the resources that will be required and output / impact indicators that can be used to measure success or failure of the programme;

(ii) Cost-Benefit Analyses (CBAs) should be conducted for all significant programme proposals, both current and capital, and should be routinely published. All such analyses should routinely include the true cost of the proposed spending including capital, accommodation, overheads and accrued pension costs; and an assessment of other lower-cost means of achieving the same policy objective;

(iii) All capital projects above €30m should be subject to a ‘look-back’ evaluation within a reasonable period of their entry into use to check realised costs and benefits as against the original CBA projections;

(iv) All expenditure programmes should have a ‘sunset clause’ after which the scheme is wound down, unless the programme shows clear positive results on the basis of a rigorous evaluation exercise, including VFM & Policy Reviews. We need a decisive move away from the existing system whereby resources are assigned once and, in effect, retained indefinitely with little regard to results or alternative priorities;

(v) Particularly in the case of grant schemes, the cash allocation should be capped at the outset, so that the Exchequer exposure is limited to the amount envisaged by the Government. The Group understands that Government accounting arrangements already provide for “cash limiting” of this nature;

(vi) The annual Estimates of Expenditure should be produced on a programme-by-programme basis, fully consistent with the Annual Output Statements and our proposed Public Service Performance Charter, with full allocation of administrative and staffing costs; and

(vii) The competency should be developed to allow expenditure programmes to be challenged and tested, on the basis of independent and publicly-available evaluation of value-for-money and effectiveness. This competency should be developed through enhanced VFM & Policy Reviews that are more tightly woven into the resource allocation process, and through a stronger role for an independent body such as the Comptroller & Auditor General (C&AG). In the same way that the Appropriation Accounts are audited by the Comptroller & Auditor General (C&AG) each year, the Annual Output Statement should be subject to independent audit / verification by the C&AG in respect of the outputs actually delivered in the previous
year; the efficiency and effectiveness of delivery; and the quality and measurability of the outputs selected for the year ahead. This initiative would also facilitate the Dáil Committees in the exercise of their independent role in holding Government expenditure up to scrutiny.