Nancy Cartwright is a philosopher of science whose work on causality is particularly relevant to social and behavioural scientists working on policy questions. Among many well-known works, her highly-cited book "Hunting Causes and Using Them: Approaches in Philosophy and Economics" is a fascinating discussion of the many different ideas of causality that are at play when we talk about relationships between social and economic quantities. More recently, many readers will recall her paper on understanding and misunderstanding of randomised trials with Angus Deaton which has been widely cited in the discussions around the use of RCTs in evidence-based public policy (See reply from Guido Imbens for a sense of the parameters of the discussion). Cartwright is obviously working within technical literature but is far from an obscurantist and I believe any social science researcher would benefit from engaging with her work. A recent paper below "Rigour versus the need for evidential diversity" is particularly relevant to many current debates about how behavioural and social science is being embedded into public policy organisations. It gives always a strong sense of points made across her work on the limits of randomised trials, the nature of causality, and the tensions between the development of generalised theories and context-specific usable knowledge.
This paper defends the need for evidential diversity and the mix of methods that that can in train require. The focus is on causal claims, especially ‘singular’ claims about the effects of causes in a specific setting—either what will happen or what has happened. I do so by offering a template that categorises kinds of evidence that can support these claims. The catalogue is generated by considering what needs to happen for a causal process to carry through from putative cause at the start to the targeted effect at the end. The usual call for mixed methods focusses on a single overall claim and argues that we increase certainty by the use of different methods with compensating strengths and weaknesses. My proposals instead focus on the evidence that supports the great many subsidiary claims that must hold if the overall one is to be true. As is typical for singular causal claims, the mix of methods that will generally be required to collect the kinds of evidence I urge will usually have little claim to the kind of rigour that is now widely demanded in evidencing causal claims, especially those for policy/treatment effectiveness. So I begin with an exploration of what seems to be intended by ‘rigour’ in such discussions, since it is seldom made clear just what makes the favoured methods especially rigorous. I then argue that the emphasis on rigour can be counterproductive. Rigour is often the enemy of evidential diversity, and evidential diversity—lots of it—can make for big improvements in the reliability of singular causal predictions and post hoc evaluations. I illustrate with the paragon of rigour for causal claims, randomised controlled trials (RCTs), rehearsing at some length what they can and cannot do to make it easier to assess the importance of rigour in warranting singular causal claims.
From my perspective, there is clearly related discussion as to the pressure to search for generalisable and robust causal parameters in areas where these are exceptionally hard to pin down is placing on career-track social scientists. It has hung like an acid cloud over many sub-disciplines in the last 20 years and the fall-out from some of it has been very painful in some cases. It is clear that the emerging empirical behavioural public policy will develop countervailing forces, including moves towards open-science frameworks that don't prioritise statistical significance in the publication process, greater focus on heterogeneity, mixed methods frameworks and diversity of evidence as outlined in the Cartwright paper, and more tolerance of ambiguity and a role for deliberation.
No comments:
Post a Comment