Below are the speaking notes from an informal talk I gave at Geary Institute on April 21st 2021 summarising work we have done over the years using day reconstruction and similar methods to study well-being and behaviour.
Overview
Naturalistic monitoring methods such as diaries, time-use surveys, experience sampling, etc., measure behaviour and well-being in everyday life. We have heavily worked on these methods over the years with a view to integrating them into behavioural decision making and behaviour welfare policy frameworks. The work has been conducted across UCD, Stirling, and now LSE in a highly collaborative way across a number of people. This is a short personal sketch of naturalistic monitoring work we have conducted at the Institute. It is intended to stimulate discussion on where we might go. It is not intended for wider circulation. The work we conducted using these methods in this group has led to several serious publications that have been cited to a reasonable degree. I hope we can use this discussion to think about what we have learned and where this work can go.
DRM Studies at UCD
I checked my UCD email archive. The first mention of day reconstruction in our setting was 2007. I remember Arie Kapteyn recommending we look at when I visited RAND. I can remember it as a sunny day up on the roof of RAND in Santa Monica. I can even remember I got slightly sunburned. We had been doing a lot of surveys mostly in student populations in the first couple of years of the Geary group. The work from that time tended to take the form of large batteries of health and psychometric scales combined with various types of measurement innovations e.g. anchoring vignettes. I was very excited by Arie's suggestion of the use of DRM as a measure of welfare.
I have quite a few emails in my UCD account badgering a range of research assistants to help me source literature and proof-read various proposals. I can remember quite clearly being very enthusiastic about the DRM. The first main impression was how informative the positive and negative well-being graphs. I had not seen anything like them before. Dave Comerford and I had been reading a lot about things like the distinction between experienced and decision utility and various concepts like that. I had also started to publish papers on determinants of well-being and mental health. The second impression was how cool it was to have people like Kahneman, Schwartz, Stone, Shkade, and Krueger all working together. I got to meet all of them with various degrees of intensiveness over the years and have heard a lot about how the paper developed (including the "read-out-load" sessions in Kahneman's house).
Michael Daly and I began to seriously discuss developing this research here in the summer of 2007. At that point we had a very active research group working across a lot of projects. Colm Harmon was the main senior UCD Professor involved but was running the Institute as a whole and my role was to develop and run the projects. It was a bit intense to be honest but the atmosphere in the group was fantastic and nearly everyone from that period keeps in touch to some degree. The team met every Monday morning to review where we were across a wide range of projects. Some of them were very report-heavy government projects and I had a feeling at the time that if we were to get anything out of this, we needed to start doing something that spoke to international audiences, and something distinctive, transdisciplinary, and potentially transformative. Michael was doing his PhD with Mac MacLachlan and particularly interested in self-control and regulatory processes. Lorna Sweeney, one of our first research assistants, introduced him to me and Colm and things spiralled quite quickly.
Our first attempt to get serious external funding for the DRM was a 2007 SFI application - the reviews from that are attached to the email. I didn't really realise at the time that SFI had no interest in social science and it was disappointing to get the final rejection but the initial reviews themselves were good. We had a number of funding setbacks on DRM over the years. One particularly painful one was the retraction of a 5 million euro HRB award that a team of us has put together in 2008. This was one of four proposals selected from a huge pool and we were notified of funding success. The grant was later turned down on the basis that the funder had simply run out of money.
At this
point, we were still very much interested in DRM as a tool for eliciting
well-being. Papers like “Back to Bentham” were being discussed a lot at the
time. Kahneman and others had also been widely commenting that life
satisfaction measures might be contaminated by representativeness biases. In
general, there was a strong atmosphere around the potential for elicitation of
episodic well-being to contribute to the development of well-being research. I
am not sure how explicitly conscious I was of this at the time but of course
one of the main historical attempts to do something like this was in
Edgeworth’s Mathematical Psychics, which is one of the most influential
economics books in history for a wide range of reasons. Edgeworth studied in
TCD and would have lived pretty much beside the buildings where we conducted
the 2008 study. It is a nice historical footnote and I think it is fair to say
that the spirit of what we were trying to do up to that point was the fairly
innocent attempt to build a hedonometer that could be used to integrate regular
well-being assessment into public policy.
Heckman was very interested in what we were trying to do. We had some fascinating interactions with him. He was quite well known at the time for sending quite terse emails on his blackberry. One I remember is that I received this reply from him on 25th February 2008 on his receiving the first draft of our proposal "Discounting and self control are right on target I have little sympathy for happiness research by the way Kahneman won his prize in 2002 not 200o". It was an interesting life lesson. If you are going to get a Nobel laureate to comment on your proposal don't have a typo in the document claiming someone else won the Nobel in the year they won it.
2008 UCD/TCD Study
Through
Geary and UCD, we were given approximately 10k to develop the idea. Michael and
I were the main drivers of the scientific aspects of the study, with Michael in
particular driving aspects relating to self-control and biological functioning.
Through Heckman, we got what many projects sometimes need i.e. a deadline. We
were asked to contribute to a special session at the 2008 European Economics
Association in Milan, which meant that the data and results needed to be
developed over the first half of 2008.
The first key paper from this project was published in the Journal of European Economics Association special issue below (https://media.wix.com/ugd/66e652_b6d63c08efa44781ab1c45e28284875d.pdf). We presented it at the 2008 European Economics Association in Milan. It was a Heckman-organised session and the room was completely packed. I am not sure everyone was convinced by what we were doing but it was certainly new. I don't think even now anyone has managed to integrate bio-markers, economic preference elicitation, and naturalistic monitoring together into an empirical framework like this. We subsequently published further papers with the data in Health Psychology (https://media.wix.com/ugd/66e652_76984f3cabdd46ce9466b2bb03fd579f.pdf9) and Emotion (https://media.wix.com/ugd/66e652_05ebf39f7e3e43fd898750a55bd39148.pdf), and Journal of Behavioural Medicine. The Emotion and Journal of Behavioural Medicine papers were very much driven by Michael, as a core part of his PhD dissertation. Michael also won a Fulbright fellowship to work with Roy Baumeister for the later part of his dissertation, and RB is one of the collaborators on the JBM paper. Interestingly one of the collaborators on the bio-papers was Peter Doran, head of the Clinical Research Centre, a joint initiative of UCD and its two major teaching hospitals, the Mater and Vincents. We worked between Geary and CRC across several projects and developing something like this still remains a big idea for public policy. To this day, the UCD/TCD study is one of the most productive projects I have worked in terms of publications but more importantly in terms of novelty and ambition. It also shows that you can do a lot with a relatively small budget if you really have the right set of ideas. We could definitely have done more and I am always a little regretful we weren’t more clever in shaping the audiences for the paper but in general, it was a really successful project and wonderful to work on.
Stirling/UCD Studies
The Stirling research group began in 2012. Michael and I worked mostly on longitudinal studies of the relationship between childhood psychological traits and economic outcomes. For the most part, there was not a great deal of new development of naturalistic monitoring work in Stirling during the first few years. The UCD DRM papers started to be published and we presented them in various formats. One key development was working with Orla Doyle on integrating DRM into the evaluation of a major early childhood intervention initiative. The paper was eventually published in PLOS-one (Doyle et al 2017). As far as I am aware it is the first paper to integrate day reconstruction type measures into a policy field experiment. (https://journals.plos.org/plosone/article/authors?id=10.1371/journal.pone.0169829).
The main idea during this period was that DRM could be augmented to study decision making processes directly. Leonhard Lades and I came in contact through the broader IAREP network. Leo I think believed that the format we set up in the JEEA paper was potentially very useful for a range of questions he was interested in. The key paper from this collaboration published in the Journal of Behavioural Decision Making is below (https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2031). Through this period, we also developed a set of applications of this work to areas such as food choice and internet usage.
The DRM
work interacted very well with the development of our MSc programme in
Behavioural Science at Stirling and several MSc students worked on this area.
It also provided material for really interesting lectures in both UCD and
Stirling, and eventually LSE. It is a nice output of the work that almost
every MSc student in Stirling and UCD, and now LSE has had strong exposure to
these methods and the thinking processes behind them.
We set up
the new UCD group in late 2016. From the start the development of naturalistic
monitoring was a key aim of this group. Leo and I developed a number of talks
and lectures in this area that were part of the development of this group.
Lucie Martin worked with us as an MSc student and research assistant and
developed a review of the area that was published in Behavioural Public Policy
(https://www.cambridge.org/core/journals/behavioural-public-policy/article/informing-behavioural-policies-with-data-from-everyday-life/9BA82DE7F9DEB3B0FE86561A7455C81E). One
major success story was the award of the MSCA fellow to Kate Laffan for the
project MINDTHEGAP which brings naturalistic monitoring into the study of
intention-behaviour gaps in core environmental domains (https://www.mindthegapresearch.com/). Diane
Pelly also joined our team to develop DRM methods to understand a range of
questions at the intersection of behavioural science and organisational
culture. Her PhD proposal in this area was recently awarded IRC funding and she
is currently in the process of developing a major RCT study with Orla. We also
have a paper together (Pelly et al 2021) on the well-being impacts of covid-19
on home-workers. More generally, we
applied naturalistic methods in a range of key settings, including in the
context of Single Use Plastic Consumption and in a particularly salient case of
adjustment to the pandemic situations in Ireland.
A key application of DRM was during the first wave of the covid-19 pandemic in Ireland. I was part of the NPHET behavioural change committee of the overall response. This was exceptionally intense and we were conducting studies on a more-than-weekly basis. We received a small amount of funding to conduct two waves of DRM. The first one was conducted in April and later published in the British Journal of Health Psychology. The graphs were made by Leo and are incredibly stimulating to look at. Naturalistic monitoring is now fully integrated into the Irish government covid response, including a bi-weekly study conducted by the ESRI Behavioural Research Unit. In general, it was an important and innovative addition to the covid response in Ireland, and one that we should reflect further on with regard to ongoing pandemic response and future emergencies.
#LSE/UCD Studies
So far, not much has changed since I have moved to LSE. We have a number of ongoing projects that are being developed. One aspect that is developing is to incorporate ethnographic designs into DRM measures of preference. Several students are interested in working with these methods for their MSc dissertation. I am looking forward to seeing the direction that some of this takes, and how it links back to work we have been conducting in Dublin.
#Filedrawer: We do not have a huge file drawer. In general, I don't think there is anything we have published that would have problems from a publication bias perspective. The key UCD/TCD study was fully published across several papers. My sense with DRM data though is that it exhausts your ability to analyse it. The data from this original study could probably be analysed in many different ways. We also conducted a follow-up study in one of the resident halls that was not published. Leo and I conducted a follow-up study to the JBDM paper that has not been published yet but does not change the core findings of the paper. We also incorporated a short DRM in the second wave of the Irish Survey of Health, Aging, and Retirement, which has not been used.
Conclusion
I will use the end of the paper with Lucie and Leo to summarise where we could go with some of this work.
Behavioural economics shows that people are boundedly rational, have limited willpower and do not always act in ways economics textbooks would suggest (Dhami, 2016). These deviations from rationality and dynamic consistency are often systematic and predictable, as shown repeatedly in laboratory experiments (Camerer et al., 2004). Such findings have started to change economic theory (Rabin, 1998) and have substantially reformed policy-making worldwide, particularly in the UK, through the foundation of behavioural insights teams and nudge units (Jolls et al., 1998; Thaler & Sunstein, 2008; Halpern, 2015). The evidence on which many of these behavioural interventions rely often comes from laboratory environments (which often put study participants into rather artificial decision situations) and RCTs (which provide information about what works, but not about the underlying mechanisms). This paper discussed how a popular naturalistic measurement tool, the Day Reconstruction Method (DRM), can be used to inform behavioural public policies by providing mechanistic evidence on how people make decisions in the real world. We suggest that the DRM is a valuable addition to the behavioural scientist's toolbox and can complement ordinary surveys, observational data, laboratory experiments and RCTs. The key benefit of the DRM, which sets it apart from alternative approaches, is that it allows for measuring decision-making in naturalistic, everyday contexts. It is thus a method that helps to quantify the extent to which behavioural biases change our behaviour in the real world and to identify where, when and why decision-making biases occur. The DRM can show, for example, whether there are correlations between biases and simultaneous situational factors such as location, activity, social interaction partner and internal state. Measuring everyday contexts and their effects on decision-making can also help to design better behavioural policies that change the choice architecture in order to nudge people to make better decisions. Such behavioural public policy interventions should be informed by domain-specific naturalistic monitoring studies in which detailed information about a particular type of phenomenon is elicited and where domain-specific context variables can be identified. For future research, there are several potential applications of the DRM in behavioural science and behavioural public policy. The method can be used to measure the prevalence of almost any behavioural concept in every domain of life. It can measure, for example, in which real-life situations people are particularly risk or loss averse, or similarly examine the influence of everyday anchors, defaults and social norms or identities on everyday economic behaviour. The key challenge for these future studies is to design survey questions that are as similar as possible to the concepts usually identified in decision-making experiments. For several concepts (e.g., risk aversion), verbal survey questions that measure individual differences have already been designed (Weber et al., 2002). Future research can adapt these questions to relate them to intra-individual changes that can differ across situations in everyday life. Such studies will then be able to quantify how prevalent behavioural biases are, and also explore in what contexts these biases are particularly likely to arise. A key challenge for future research is to integrate DRM studies in causal designs. Since the DRM is a survey that can be completed in one sitting, it can be easily added to existing RCTs. DRM studies also lend themselves well to evaluating large policies where DRM data from before the policy implementation can be compared to DRM data gathered after the implementation. Another branch of future research should deal with methodological issues. For example, different versions of the DRM (full versus abbreviated, online versus analogue, one day versus multiple days, different reinstantiation procedures, etc.) should be compared in order to identify the effects that design choices have on participants’ response patterns. It will also be important to further test the reliability, validity and accuracy of DRM data by comparing it with experience sampling data. Cognitive testing in interviews and focus groups should be conducted to make sure that the question wording used does not confuse the participants. Moreover, filling out the DRM itself can change behaviour, and the potential of the DRM as a behavioural intervention should be explored. If we better understand these methodological issues, DRM studies measuring behavioural concepts could be integrated into existing large-scale, nationally representative time use surveys. This strategy would help us to gain a better understanding as to how individuals from different sub-populations differ in terms of the decisions they make in their everyday lives.
More broadly, there are a few thoughts we are working through:
The project to validate DRM with ESM data is still one that is very valuable. Though not necessarily for the reasons you might think. I genuinely believe ESM and DRM measure slightly different types of preferences, neither of which is more valuable.
The accumulation of applications is also a goal that is worth pursuing. So far, we have conducted domain specific studies in areas such as workplace behaviour, single-use plastics, internet use, environmental behaviours, and meat consumption.
Leo and I continue to work on the potential for Decision Day Reconstruction be a coherent tool for behavioural welfare economics in policy. The “As Judged by Themselves” paper points to a range of preferences that have a form whereby modification of choice architecture would help people resolve low-level self-control conflicts in such a way as to both improve present well-being and adherence to long-run objectives.
The
development of an economic phenomenology literature is something that I would
not have had sufficient confidence to do earlier in my career but am
increasingly interested in developing. This would treat human experience in
economic contexts as inherently interesting in itself regardless of whether it
can be used to develop welfare indicators with useful properties for
cost-benefit analysis.
Publishing work like this in journals presents challenges in terms of disciplinary alignment. Over the years we have found homes in top psychology field journals, decision making and behavioural policy journals, and wider open-access journals such as PLOS. The development of new interdisciplinary journals such as Nature Human Behaviour and Behavioural Public Policy presents another useful audience for this type of work.
Another aspect to reflect on is the extent to which we collaborate and discuss this work with wider audiences. We have been more successful in publishing this work than we have in seeing it cited across literatures. There have not been a wide range of external collaborators on the work. Roy Baumeister is the only listed author outside the collaborating institutions. This partly reflects the fact that this was a very original project pulling together a lot of strands across disciplines. More lately we have been discussing a collaboration with Will Hoffman. The development of projects at LSE will also bring in a set of collaborators. It would be good to discuss a workshop to bring our work in contact with some of the influences listed below and others people have in mind.
References
Crowe, E., Daly, M., Delaney, L., Carroll, S.,
& Malone, K.M. (2019). The intra-day dynamics of affect, self-esteem,
tiredness, and suicidality in Major Depression. Psychiatry Research, 279,
98-108. doi: 10.1016/j.psychres.2018.02.032
Daly, M., Delaney, L., Doran, P., & MacLachlan,
M. (2011). The role of awakening cortisol and psychological distress in diurnal
variations in affect: A Day Reconstruction Study. Emotion, 11, 524 - 532.
Daly, M., Delaney, L., Doran, P., Harmon, C.,
MacLachlan, M., & Daly, M. (2010). Naturalistic monitoring of the
affect-heart rate relationship: A Day Reconstruction Study. Health Psychology,
29, 186 -195.
Daly, M., Harmon, C., & Delaney, L. (2009).
Psychological and Biological Foundations of Time Preference. Journal of the
European Economic Association, 7, 659 - 669.
Daly, M., Baumeister, R.F., Delaney, L., &
MacLachlan, M. (2014). Self-control and its relation to emotions and psychobiology: evidence from a Day
Reconstruction Method Study. Journal of Behavioral Medicine, 37, 81 - 93.
Delaney, L., and Lades, L. K. (2017). Present Bias and Everyday Self-Control Failures: A Day Reconstruction Study. Journal of Behavioral Decision Making, 30(5), 1157–1167.
Doyle, O., Delaney, L., O’Farrelly, C., Fitzpatrick, N., & Daly, M. (2017). Can early Intervention improve maternal well-being? Evidence from a randomized controlled trial. PLOS ONE, 12, e0169829.
Lades, L., and Delaney., L (2025) “Self-control failures, as judged by themselves”, (mimeo)
Lades, L., Laffan, K., & Weber, T. (2021). Do economic preferences predict pro-environmental behaviours?. Ecological Economics, 183, 106977.
Lades, L., Kelleher, L., & Kelly, A. (2020). Why is active travel more satisfying than motorized travel? Evidence from Dublin. Transportation Research Part A: Policy and Practice, 136, 318-333
Lades, L. K., Martin, L., and Delaney, L. (2019). Informing behavioural policies with data from everyday life. Behavioural Public Policy, 1–19.
Lades, L., Laffan, K., Daly, M., & Delaney, L. (2020). Daily emotional well-being during the COVID-19 pandemic. British Journal of Health Psychology, 25, 902-911.
Pelly, D., Daly, M., Delaney, L., & Doyle, O. (under review). Worker well-being before and during the Covid-19 restrictions: A longitudinal study in the UK. PsyArXiv. https://doi.org/10.31234/osf.io/659yd
No comments:
Post a Comment