How should we evaluate policies?

My purpose in writing this post is to pick up on a debate we have had on this blog and some of us have debated on other forums in the context of the beginning of the new government. I don’t think any of us really believed that a beleagured government, knowing fully that it was on the way out and facing a major crisis, was really going to be a vehicle for serious economic policy change in Ireland at micro level. But the new government has the type of majority and popular appetite for reform that may never be seen again and it is a good time to start this discussion anew.

A key feature of an economic recovery should be a more intelligent policy design and analysis framework for assessing policy changes in areas such as health, social insurance and education. At least part of this will involve greater economic skills on behalf of existing staff, more public assesment frameworks and less direct political interference in the actual design and implementation of policies. Another part will come from a willingness to engage with developments in economics and economic methodologies, particularly those emanating from microeconomics, the half of economics that hitherto has mostly been ignored by Irish governments.

A technical question of sizeable importance is how we should best actually evaluate public policies and who should take responsibily for doing this. By public policies, I mean any action of the state, including direct provision of funds, legislation, the creation of regulatory agencies and so on. One approach relies on measuring relevant economic variables in micro-units and examining the causal effect of the change and the level of the change. Various quantities can be calculated, including the average effect of the change across the population of interest and the effect on people specifically affected by the policy.

Estimating these types of quantities is technically and practically complex. One solution that has been advocated for several decades by various researchers particularly in the US and has found new force among development economics researchers in particular is the use of Randomised Controlled Trials (RCTs) to examine the effect of policies. A large number of researchers have been advocating the use of RCTs in public policy, with their main putative benefit being that they allow the isolation of the causal effects of policy.

There has been somewhat of a backlash against the more evangelical view of RCTS. In particular, people like Heckman and Deaton have pointed out that the use of RCTS brings new problems to the table, including the potential for Hawthorne effects, the difficulty in externally applying results that come from model trials, the cost of implementing trials and so on.

It is worth the while of the new government to engage with this debate. We have talked before about the fairly limited engagement, for example, of the aid programme with developments in Economics and the international debate in development economics would certainly help sharpen the case for spending money in these areas. One thing that should be kept in mind is the potential for the rapid deployment of randomised trials in the development of the surface features of tax and social welfare policy. An increasing literature in behavioural economics is showing impressive effects of fairly cheap changes in things like the nature of forms used in different policies and default options. This is spawning an increasing literature, some of which is convincing. As the Irish government moves to policies such as automatic enrolment for pensions, it will be increasingly important to examine how the surface features of these policies will influence people’s behaviour.

A few things I would say that I believe are reasonable conclusions from the current debate:

1. Randomised Trials should not be overhyped but they have strong potential in specific areas and the government should examine this potential.

2. A greater focus on defining the outcomes of public policies and engaging in sufficient longitudinal tracking to enable assessment of these outcomes should characterise the main decisions made by the new government. The types of radical changes coming down the tracks to pensions and healthcare should be matched by an effort to increase the level of intelligence used to assess these changes.

3. While the development of an independent fiscal council is obviously a positive step, the new government should have greater engagement with developments in microeconomics, including a greater emphasis on peer-review of major policy evaluations.

4. A cultural norm needs to emerge placing greater pressure on government departments and agencies to provide solid rationales for large-scale policy changes. In the US, various legislation mandates the deployment of rigorous methodologies to assess policy changes.

5. Policymakers need to become more aware of the basic nature of modern economics. Many recent Irish policy documents are noted by the real failure to engage with the type of research that actually drives the fields under discussion e.g. the innovation taskforce document was particularly weak in its research foundations and strategic documents are routinely published that do not seem to draw in any way from what has been learned in the relevant literatures.

6. One potential solution would be the development of a fast-track economics group within the Irish civil service, something that Karl Whelan, Colm Harmon and others have spoken about. In my view, such a group would need to place a heavy emphasis on ability to synthesise economic knowledge on sectoral and specific micro-policy issues. This does not neccesarily mean recruiting outside of the existing service but it does mean breaking norms within the civil service and prioritising the development of a group who would be given a mandate to engage with the wider economics world. The lack of economic expertise in areas such as health, social protection and finance is genuinely frightening and damages the credibility of the state in a major way, something that will be increasingly exposed in a less opulent environment. The almost complete absence of anyone with a PhD Economics qualification from the Irish civil service is something that cannot continue.