Saturday, February 14, 2015

Reference bias in self-reported personality measures

From p18 of Heckman & Kautz (2013). Fostering and Measuring Skills: Interventions that Improve Character and Cognition. NBER Working Paper:

"Answers from self-reports can be misleading when comparing levels of personality skills across different groups of people. Most personality assessments do not anchor their measurements in any objective outcome. For example, the German Socio-Economic Panel (GSOEP) survey asks respondents to rate themselves on the following statement:"I see myself as someone who tends to be lazy". The scale ranges from 1 = "strongly disagree" to 7 = "strongly agree." In answering this question, people must interpret the definition of "lazy," which likely involves comparing themselves to other people. If different groups have different standards or reference points, comparing traits across groups can be highly misleading. Laziness may mean different things to different groups of people.

Schmitt, Allik, McCrae, and Benet-Mart nez (2007) administer a Big Five personality questionnaire to groups of people in a variety of different countries. Using their estimates, [the below figure] shows how OECD countries rank in Conscientiousness from high to low. The bars display the average number of hours that people work in the country. The results are surprising. South Korea ranks second to last in terms of Conscientiousness but also ranks first in the number of hours worked. South Korea is not an anomaly. Country-level reports of Big Five Conscientiousness are unrelated to the number of hours worked."

I used to live in Korea, and my impression was that Koreans are incredibly hard-working as a group, but have such high expectations of what 'hard-work' means that few individuals regard themselves as being particularly diligent.


Liam Delaney said...

Useful paper on this below Mark

See also the Gary King website on using anchoring vignettes in cross-cultural settings

Mark McGovern said...

There is also interesting recent work on this issue in relation to SWB:

Beegle, K., Himelein, K., Ravallion, M., 2009. Frame-of-reference bias in subjective welfare regressions. World Bank Policy Research Working Paper Series

Dahlin, M.B., Kapteyn, A., Tassot, C., 2014. Who are the Joneses? CESR-Schaeffer Working Paper.

Ravallion, M., Himelein, K., Beegle, K., 2013. Can subjective questions on economic welfare be trusted? evidence for three developing countries. Evidence for Three Developing Countries (December 1, 2013). World Bank Policy Research Working Paper.