Massive increases in the availability of informative social science data are making dramatic progress possible in analyzing, understanding, and addressing many major societal problems. Yet the same forces pose severe challenges to the scientific infrastructure supporting data sharing, data management, informatics, statistical methodology, and research ethics and policy, and these are collectively holding back progress. I address these changes and challenges and suggest what can be done.
I just read about a new company called Affectiva, which "grew out of collaborative research at the MIT Media Lab to help people better measure and communicate emotion."
This blog has previously discussed how human behaviour and sentiment is being analysed using data from the web: (i) Web search volume can can improve forecasts of the current level of activity for a number of different economic time series, including automobile sales, home sales, retail sales, and travel behaviour (Google research paper on "predicting the present"). (ii) What people are searching for today can be predictive of what they will do in the near future (Yahoo! PNAS paper) (iii) There have been various initiatives to make sense of sentiment data from Twitter (iv) Facebook are analysing a rich seam of social networking and affect data (v) Dan Hopkins and Gary King have developed an automated program to gather sentiment from online blogs
What makes Affectiva different to all of the above, is that (through their project Affdex), they have provided a way to collect web-data on physically expressed human emotions, not stated mood or revealed (search) preferences. Data on facial expressions and gestures — while people interact naturally with media, or other online experience — can be collected using Affdex. One could argue that this is closer to the objective truth than self-reported affect or observed behaviour.
Also, Affectiva's Q Sensor project measures galvanic skin response using a wearable, wireless biosensor. The sensor also measures temperature and activity.
ReadWriteWeb.com: "Thanks to a cooperation with Citibank and other third-party data suppliers, Bundle is able to compile detailed statistics about how Americans are spending their money. While lots of banks also compile this data, Bundle is the first service to make this data easily accessible." Using Bundle, American citizens can investigate whether their spending patterns are in line with that of other people in their age and income group in their neighborhood. This brings a new twist to ideas from behavioural economics such as reference-dependency and other-regarding preferences.
Bundle plans to update its data on a quarterly basis. It also plans to allow users to enter their exact spending habits by either entering the data by hand or by giving users the option to upload credit card statements directly. In relation to self-reported spending habits, Dave, Liam and Colm have done research on "Experimental Tests of Survey Responses to Expenditure Questions". Bundle is also set to provide budgeting advice based on its users' spending habits. Bundle is a joint collaboration between Microsoft, Citigroup and Morningstar. The site fully integrates with Facebook and Twitter. A brief video from the folks at Bundle.com is shown below.
Liam linked to an article in the Guardian earlier this week, which was all about Nudge. One comment in the article was that "while shopping, working, or even deciding on who to share their lives with, individuals are less thoughtful and less calculating than modern-day economists... typically assume." This blog-post zones in on shopping, in particular the data-analysis of consumer purchasing behaviour at Tesco. The Guardian article linked above also suggests that "any critic who points out that that's hardly news to the women...(and) the men at Tesco... is spot on." Indeed, Tesco have been conducting interesting micro-level analysis on individual behaviour for many years now.
An informative article on this topic was written by Jenny Davies in the Sunday Times last year. According to Davies, Tesco gets its data from its loyalty clubcard scheme; this was launched 15 years ago with much fanfare - the advert below may jog memories for some readers. Davies also informs us that around this time last year, Tesco was tracking "the shopping habits of 16 million families across Britain, delivering an extraordinary insight into their lives — not only for itself but for companies such as Coca-Cola, Nestlé and Unilever, which buy the rights to the data." Readers in the Republic of Ireland might also remember that the Tesco Clubcard was launched there on the 13th. Oct 1997. To date almost 800,000 members have joined in the Republic.
Jenny Davies also tells us that: "Each bill detailing every item in a customer’s shopping basket is logged in a data centre in London Docklands and decoded by Dunnhumby, the marketing firm that is in charge of the scheme. It has to process 100 baskets a second — six million transactions a day. This helps Tesco to decide which products should go on to the shelves at what times, and in early trials it increased sales by as much as 12% in some of the supermarkets." According to the Guardian (in this article), the power of the clubcard was demonstrated in 2009, "when Tesco harnessed the card's database to halt the exodus of shoppers to cheaper retailers because (of) the recession, by doubling the points available to shoppers."
In a blog-post on Tesco data from two years ago, Tony Hirst desribes the early analysis conducted by Dunnhumby, and how this has changed over the last 15 years. A couple of months ago, Dunnhumby (and its recently departed co-founders) were profiled in the Guardian. The article says:
According to company lore, there was a 30-second silence after Humby presented the initial trial's results to the Tesco board, until the then chairman, Lord MacLaurin, declared: "What scares me is that you know more about my customers after three months than I know after 30 years."
One question that readers might have is: what's in it for club-card holders? According to Tony Hirst, a good place to get an answer to this question is the book: Scoring Points: How Tesco Continues to Win Customer Loyalty. Hirst describes the "Clubcard customer contract: more data means better segmentation, means more targeted/personalised services, means better profiling. In short, the more you shop with us, the more benefit you will accrue." According to the Marketing Week magazine, "from the day of its launch in February 1995 the Tesco Clubcard was immediately embraced by customers attracted to the 1% discount off their shopping bills. But its long term success has not been built on discounts alone, rather on the personalisation of the shopping experience."
However, perhaps the last word should go to UCD social psychologist Ken McKenzie, writing on his A Head in Business Blog: "I don’t have a loyalty card, and every time I’m in Boots, Tesco or Dunnes, and they ask if I have one, I feel a slight sense that I should justify why I don’t, as it it’s odd to not have one. And according to rational actor theory in Economics, it is odd to not have a loyalty card and avail of discounts. However, there’s a small but growing body of work in the overlapping area between Psychology and Economics that might explain why (some) people might behave like me."
However, it has only now come to my attention that Varian has a counterpart at Yahoo! The economist in question is Preston McAfee, on leave from the position of J. Stanley Johnson Professor of Business, Economics, and Management at the California Institute of Technology. At Yahoo!, Professor McAfee is Vice President and Research Fellow at Yahoo! Research in Burbank, CA, where he leads a group focused on microeconomics research. Here is a list of recent publications from the Microeconomics and Social Systems cluster at Yahoo!
Professor McAfee wrote Introduction to Economic Analysis, a free, open-source text that spans both principles and intermediate microeconomics. In 1994, the FCC in the USA auctioned access to a number of radio frequencies for new communications services, using an auction designed by Paul Milgrom, Robert B. Wilson, and Professor McAfee, and raised over $17 billion. This auction design was copied around the world. McAfee, Milgrom, Wilson and John McMillan (1951-2007) formed a company, Market Design, Inc., that advises governments on how to maximize the return from sales of radio frequencies, mineral rights, airports, and other assets.
Returning to Yahoo!, The Register magazine describe the company's Right Media exchange — a display advertising marketplace that matches advertisers with publishers and ad networks — as (by one measure) the largest exchange in the world, running over nine billion auctions each day. Finally, the Yahoo! Advertising Blog is also an interesting read; the current post - Mad Men No More - features a discussion by "advertising’s new guard" on how they are re-defining the industry.
1. President Barack Obama chose Austan Goolsbee to succeed Christina Romer as the head of the U.S. Council of Economic Advisers. Here, the Wall Street Journal do a profile of Goolsbee.
2. The Guardian: a "nudge unit" set up by David Cameron in the Cabinet Office is working on how to use behavioural economics and market signals to persuade citizens to behave in a more socially integrated way.
4. Greatest Good: "a unique firm formed with the goal of applying rigorous, cutting-edge data analysis and economic methods to the most salient problems of business and philanthropy." Founding partners include Steven Levitt, Gary Becker, Daniel Kahneman and John List. Affiliates include David Laibson, Emily Oster, Steven Pinker and Richard Thaler.
5. The U.S. National Commission on Fiscal Responsibility and Reform. They have a separate mandate to the Congressional Budget Office. "The Commission is charged with identifying policies to improve the fiscal situation in the medium term and to achieve fiscal sustainability over the long run."
6. A fascinating read for any Ph.D. student in Economics, or Ph.D. economist: 'Market Structure in the Production of Economics Ph.D.s'. Frank A. Scott, Jr. and Jeffrey D. Anstine; Southern Economic Journal Vol. 64, No. 1 (Jul., 1997), pp. 307-320.
8. University Attendance Scanners: "Northern Arizona University has installed electronic devices that record student attendance in an effort to boost freshmen grades and lift lagging graduation rates. But some students say the monitoring makes them feel less independent." (Southern California Public Radio).
2. Retention Rates of Pupils in Second-Level Schools: 1991 to 2001 Entry Cohorts. These data are aggregated but provided on a county by county basis. The overall Leaving Certificate retention rate has increased in the five years between the 1996 cohort and the 2001 cohort: from 81.3% to 84.7%.
3. The State Examinations Commission website has a statistics page providing Leaving Certificate and Junior Certificate grade information for every subject, and by gender, for the years 2000-2010.
Students and researchers should keep a close eye on the evolving data infrastructure at the Institute for Health Metrics and Evaluation at University of Washington. The Maternal Mortality data page here is an example. Many of these data-sets were constructed for large scale evaluations of public health initiatives developed with a view to the Millenium Development Goals. They are very useful resources also for development economists and do not yet seem to have been widely used in this area. They are also, for the most part, downloadable for free in a "one-click" excel format. The growing publication list of this Institute is also worth looking at carefully.
I have talked a lot about the Dutch LISS Panel as being one of the world's most useful data resource for social science. It has been nominated for a Dutch award for data dissemination. Along with it was nominated the eventual winner, the Historical Sample of the Netherlands. This is a mindblowing data infrastructure. From their website "The HSN offers a representative sample of about 78,000 people born in the Netherlands during the period 1812-1922. The HSN-database containing individual life-courses is a unique tool for research in Dutch history and demography." It is well worth a browse through the projects and data that is linked on their site.
The following project is one example of how they are using the data.
The project 'Early Life Conditions, Social Mobility and Longevity' (ESM) is an international project, supervised by prof. George Alter (Indiana University, Bloomington), and for the main part financed by the National Institute of Aging (National Institute of Health, Washington D.C.). The Dutch part is carried out by dr. F.W.A. van Poppel, from the Netherlands Interdisciplinary Demographic Institute (NIDI). The cooperation with the HSN consists of building the database 'Inequality and longevity from a life-course perspective: the Netherlands 1850-2000'.
Based on this database the long-term effects of social, economical and familial conditions during childhood on mortality in later life will be studied. Using population registers, personal cards and records of the civil registration, life courses and family compositions will be reconstructed for about 7000 children, born between 1850-1922 in the provinces of Utrecht, Zeeland and Friesland. Context variables, like the regional level of water supplies, mortality and family income, are also part of the analysis.
Hugh Linehan argues for greater transparency with data that is currently subject to Freedom of Information requests. He mentions the website http://thestory.ie/ which is “… dedicated to sharing documents, combing and combining data and promoting transparency in public life: An experiment in journalism and crowdsourcing hoping to shed light on the government. If you’re spending the Irish taxpayers’ money, you’re on the radar.”
Socrates talked about the unexamined life not being worth living. Developments in wearable data collection and feedback technologies take this to a new extreme
Google Public Data has been expanded. Below are charts based on the minimum wage data-set for European countries (for a random selection of countries). Unsurprisingly, Ireland is at or near the top in both cases. It is also possible to take logs of these series. If, like me, you want to read more about where this data comes from, here is the link to the Eurostat page. Scroll along the charts to see wage levels at any point in time (other countries fade out).
1. National minimum wages in Euros per month before deduction of income tax and social security contributions. Not adjusted for inflation. In euro.
2. National minimum wages in Euros per month before deduction of income tax and social security contributions. Not adjusted for inflation. In terms of purchasing power parity.
1. MySpace has allowed a large quantity of bulk user data to be put up for sale: user playlists, mood updates, mobile updates, photos, vents, reviews, blog posts, names and zipcodes. Friend lists are not included.
2. InfoChimps is a bulk data marketplace with more than 5000 data sets in its catalog so far. The vast majority are free.
3. A Mulley Communications/NCI study shows that people do not pay much attention to more than three results on a Google search result page. Also that the ads on the right hand side of the results page are barely looked at.
4. Here is a video of heatmaps being generated based on eye movements.
5. It's worth looking at data-series on capital expenditure: in this case in the UK. A pick-up in this series should lead improvements in employment figures.
6. Ellerdale, still in alpha testing, tracks data sources from around the web, primarily Twitter, and examines what topics are being discussed. It then organizes these conversations into categories like "people," "sports," "politics," "music," "television," and more.
7. The most rented movie in Chicago last year? "The Curious Case of Benjamin Button." This app on the NTY website allows for examination of Netflix rental patterns, neighborhood by neighborhood, in a dozen cities.
8. Netflix have announced that they canceling plans for a second Netflix Prize contest, one that would have involved the release of more information than the first.
9. Privacy concerns were an issue in the Netflix decision. Among the first to draw attention to the issue was University of Colorado law professor Paul Ohm, who said: "Researchers have known for more than a decade that gender plus ZIP code plus birthdate uniquely identifies a significant percentage of Americans (87% according to Latanya Sweeney's famous study)."
10. Here's a recent paper by Ohm on re-identification of individual data.
11. The deadline for Facebook's Ph.D. Fellowship Program has passed, but it's interesting to note that they have a particular focus on Internet Economics.
1. "What could be more baffling than a capitalist corporation that gives away its best services, doesn't set the prices for the ads that support it, and turns away customers because their ads don't measure up to its complex formulas?". Read how economics underlies every aspect of the Google business model: here in Wired.
2. AdWords is a pioneering variation on a second-price auction.
3. AdWords was such a hit that Google used auctions to place ads on other websites: AdSense.
4. Hal Varian, Chief Economist at Google, has been mentioned on the blog before: here and here.
5. "But the really gutsy move," Hal Varian says, "was using it in the IPO." In 2004, Google used a variation of a Dutch auction for its initial public offering.
6. The Google equivalent of the Consumer Price Index is called the Keyword Pricing Index. Here's a link about Fathom Online's version. Examples of very competitive keywords are 'flowers' and 'hotels'.
7. Quality Scores are important: a penalty is invoked when the ad quality is too low. In such cases, the company slaps a minimum bid on the advertiser.
8. Hal Varian says: "The people working for me are generally econometricians—sort of a cross between statisticians and economists". He's currently hiring a senior economist. The London office also has other opportunities.
4. The Collaborative on Academic Careers in Higher Education (COACHE)
5. Seeking Alpha on last Thursday's U.S. jobless claims report; the story features some graphs from Blytic.com that are worth checking out
6. The Smart Data Collective: The Data-Driven Enterprise Community; their blog recently featured a critical review of McKinsey's guide to behavioural economics for marketers
7. Intelligent Enterprise: "Possibly the most important factor influencing the spread of predictive analytics is the growing popularity of R... Vendors, including IBM SPSS, Information Builders and SAS are incorporating R." More here.
1. A new report from the European Centre for the Development of Vocational Training: "The skill matching challenge: Analysing skill mismatch and policy implications". HT: Michael Egan. [The report is largely based on the contributions of Nigel O’Leary and Peter Sloane (Welmerc, School of Business and Economics, Swansea University), Seamus McGuinness and Philip O’Connell (ESRI – Economic and Social Research Institute, Dublin) and Kostas Mavromaras (NILS – National Institute of Labour Studies, Flinders University, Adelaide, Australia).]
"With global competition increasing, demographic change unfolding and rapid technological change intensifying, skill mismatch has come to the forefront of Europe’s policy debate (European Commission, 2008). Skill mismatch refers not only to skill shortages or gaps, but also to qualifications, knowledge and skills exceeding job requirements."
2. The U-S unemployment rate fell from 10 to 9.7 percent in January, but a Gallup poll out yesterday found that nearly one out of every five members of the country's workforce is underemployed. That's about 30 million Americans who are without jobs or unable to find full-time work.
3. NYT: "Economists fear that the nascent recovery will leave more people behind than in past recessions. Roughly 2.7 million jobless Americans will lose their unemployment check before the end of April... unless Congress approves the Obama administration’s proposal to extend the payments"
While Google's Fusion Tables may not appeal to professional researchers who are conscious of data-security, there are some useful features which may appeal to beachelor's and master's students working on group projects. These include:
- Upload small or large data sets from spreadsheets or CSV files. - Visualize your data on maps, timelines and charts. - Pick who can access your data; hide parts of your data if needed. - Merge data from multiple tables. - Discuss your data with others. Track changes and discussions.
Day-of-the-week effects have been mentioned on the blog before, including research by Gerard O Neill from Amarach Consulting. A recent article in the New York Times reports that "there is a 9.7 percent increase in happiness on Fridays compared with the worst day of the week, Monday. That is among the discoveries made by Facebook researchers with access to two years of anonymous “status updates” from 100 million users in the United States."
Updates from Facebook's data-team can be accessed here. The Top 15 status terms of 2009 can be viewed here. Analysis of maintained relationships on Facebook can be viewed here. With more data than ever available through the rise of Web 2.0, the Flowing Data blog suggests that we may see the rise of the "data-scientist".
The debate into the Irish recession has naturally focused on the banking system and the fiscal adjustment. Ireland ran a large bubble in property for at least five years that added to revenue, inflating both the public sector coffers and the banking systems balance sheets. When the financial crisis hit, the vulnerability of this situation became rapidly apparent and we have witnessed an astonishing increase in unemployment as well as collapse of our banking system and a need for rapid fiscal adjustment that has, among other things, reduced take-home pay of public servants.
One real shame about our current situation is the lack of data to monitor how it is actually affecting people. The current micro-data infrastructure of Household Budget Survey, SILC, SLAN, SHARE, QNHS and others is simply ill-equipped to examine events that occur at this level of frequency. Thus, we simply do not know how households are smoothing consumption through this time period. For example, we know that many people have taken drastic reductions in their housing wealth but do not know how this actually affects their day-to-day lives with respect to availability of credit. Nor do we have any real sense of the extent to which certain groups are facing hard liquidity constraints in ways that may lead to real consequences for health and life outcomes. Anecdotally, we are hearing reports of paid doctors visits declining as the recession bites but have no sense of whether people's health is being placed in jeopardy. Similarly, preliminary data suggests an increase in suicide rates, but we are unable to examine the mechanisms through which the economy impacts on psychological distress. Savings have increased sharply as has consumption has declined but the academic community has no way of providing any micro-founded explanation of why this is happening.
We have been collecting monthly panel data here to track things like risk perception but the research is still too early stage to provide sufficient sample sizes for sub-groups to answer the questions above. My personal new year's resolution is to get this to a state where we can start engaging properly with policy in Ireland. High-frequency (i.e. monthly) data delivered quickly to academic researchers in a manner comparable to other countries is the only solution to the massive knowledge hole that exists in Ireland at present. We need to speed up the delivery of data so that papers can be written that are scientifically robust and timely enough to actually enter into the debate.
