A Less Controversial 3ie Replication Study: The Impact of India's JSY Conditional Cash Transfer Programme

The impact of India's JSY conditional cash transfer programme: a replication study

3ie Replication Paper 6, 2015

http://www.3ieimpact.org/publications/3ie-replication-paper-series/3ie-replication-paper-6/

 

Natalie Carvalho and Slawa Rokicki

Abstract

Conditional cash transfer programmes are becoming increasingly popular in low- and middle- income countries, with the goal of improving access to health and social services and reducing inequities in access and outcomes for the poor and marginalised. India’s conditional cash transfer programme, Janani Suraksha Yojana (JSY), established in 2005, is one of the largest such programmes in the world. Along with small payments to community health workers, it provides financial incentives to pregnant women to encourage them to deliver in health facilities. Lim et al.’s Lancet article, ‘India’s Janani Suraksha Yojana, a conditional cash transfer programme to increase births in health facilities: an impact evaluation’ (2010), was the first formal statistical impact evaluation of the programme across the whole of India. This replication study, through robustness checks and additional model specifications, re-examines this recent work on the effect of financial incentives for women through the programme on reproductive health coverage indicators and perinatal and neonatal mortality. Of three analytic approaches taken by Lim et al. (2010), this replication focused on exact matching analysis, using data from round three of India’s district-level household survey (DLHS-3). We found the original authors’ results to be replicable and robust to various changes in model specifications and analysis. We were able to replicate quite closely the national and subnational results reported by Lim et al. We conducted several additional analyses as robustness checks including alternative matching estimators, analyses to account for differential implementation of the programme and random effects models to examine state-level heterogeneity in results. We found meaningful heterogeneity across states and districts in the effects of JSY on reproductive health coverage indicators and mortality outcomes. Accounting for state-and district- level heterogeneity has important implications for understanding the effectiveness of this programme.

Keywords: replication study, conditional cash transfers, maternal health, neonatal mortality, skilled birth attendance

Links on the replicability crisis

Following yesterday's talk by Professor Paul Mills on the replicability crisis in science, here are some links on the topic.

1. Excellent write-up of the still unfolding claims of data-fabrication concerning this Lacour and Green (2014) article, which has now been retracted by Science.

2. John Bohannon, author of "Who's Afraid of Peer Review?" (2013), has a new article out on an intentionally ludicrous study he published about chocolate and weight-loss which nonetheless generated global media coverage.

3. The article that arguably kick-started the whole replication movement is the dramatically titled "Why Most Published Research Findings Are False" by Ioannidis (2005). It is now approaching 3,000 citations.

4. See the work of the Many Labs group which replicate psychological studies across many different lab environments to test their robustness (i) Many Labs 1 (ii) Many Labs 2 (iii) Many Labs 3. There is also movement within behavioural economics to test the replicability of old economics results using prediction markets.

5. The always excellent Data Colada blog recently examined an attempt to replicate the study that inspired a TED Talk on power posing which has been viewed 26 million times.