StereoTypes

"There must be more to life...
Than stereotypes"

So goes the famous song by Blur, but this doesn't ring true for those of us with a keen interest in analysing ordinal outcome variables. According to Mark Lunt (writing for Stata.com), one common approach, known as the Proportional Odds (PO) Model, is implemented in Stata as ordered logit.

If the assumptions of the PO model are not satisfied, an alternative is to treat the outcome as categorical, rather than ordinal, and use multinomial logistic regression in Stata. It is also possible to use Anderson's "Stereotype Ordinal Regression (SOR) Model", which according to Lunt, "can be thought of as imposing ordering constraints on a multinomial model. The multinomial model provides the best possible fit to the data, at the cost of a large number of parameters which can be difficult to interpret. Stereotype regression aims to reduce the number of parameters by imposing constraints, without reducing the adequacy of the fit."

The stereotype approach is also discussed in Scott Long's book on "Regression Models for Categorical Dependent Variables using Stata" (exact page opens up), and in an epidemiology review article by Annath and Kleinbaum which takes a broad look at regression models for ordinal responses. A relevant publication by Mark Lunt (in 'Statistics in Medicine'; 2005) is available here: "Prediction of ordinal outcomes when the association between predictors and outcome differs between outcome levels".

Finally, those with an interest in anchoring vignettes may be interested in this work by Johnson that was published in Psychomatrika less than two years ago: "Discrete Choice Models for Ordinal Response Variables - A Generalization of the Stereotype Model". Johnson discusses the case of the generalized stereotype model, which includes category-specific random effects due to individual differences in response style. "...Unlike standard random utility models the generalized stereotype model is better suited for ordinal response variables and can be interpreted as a kind of unidimensional unfolding model".

Completing an Economics PhD in Five Years

Thanks to Christian for pointing out this paper published in the AER today (by Stock, Finegan and Siegfried). Endogeneity concerns aside, finishing your PhD within the designated time is positively affected by:

- larger 1st year PhD classes
- shared offices
- being male
- whether someone went to a top tier university for undergrad

Factors that have a negative effect are:

- doing your PhD with a top tier instution
- high attrition in the 2nd year
- pre-thesis research work requirement
- having an undergrad degree in economics

The authors conclude that "many considerations unique to individual students and faculty that we cannot measure—such as ambition, motivation, persistence, organizational skills, the creativity of students, and interest in students’ success as well as mentoring and motivational skills among graduate faculty—matter more than the myriad characteristics we were able to measure, which collectively account for less than 15 percent of the variation in completion among students."

Some insights on how non-cognitive personality constructs (such as ambition, motivation, persistence and organisation) apply to graduate education are provided in the Educational Assessmnet Journal (2005) by Patrick Kyllonen, Alyssa Walters and James Kaufman from the Princeton Educational Testing Service. We discussed this research on the blog before: here.

Stock and Siegfried (2006) reported on time-to-degree for economics Ph.D.'s in the United States in the AEA Papers and Proceedings. That research motivated me to consider that the duration of the Ph.D. process (or time-to-degree) may be a source of comparability problems in self-rated skills matching for Ph.D. graduates (see a previous post on skills-matching here).

The idea is that the more the individual has committed to the process of atatining a Ph.D., the more he or she will want to view the outcome of that process favourably. Taking a year longer during Ph.D. training implies a very particular opportunity cost. There is a precedent for this type of comparability-bias in the anchoring vignettes literature.

Buckley (2007) used the anchoring vignettes technique to investigate the "rose-coloured glasses" effect, which refers to parents reporting higher levels of satisfaction with a school solely or partially as a justification for the effort expended in the choice process. The analogy to 'time-to-degree' is about the amount of time expended in the Ph.D. process. (See a previous discussion of Buckley's research here).

Some Like It Hot

Paul Gottemoller and Randolph Burnside from the Department of Political Science at Southern Illinois University have written a paper entitled "Are They Still Hot?: Utilizing Feeling Thermometers and Anchoring Vignettes to Measure Affect". Paper available here. Abstract here.

The authors recount how "feeling thermometers" have been used in survey research since the beginnings of modern survey research - this is a way to ascertain individual feelings in a multitude of settings. They emphasise that it is difficult to compare one respondent’s self-ranking to another respondent’s self-ranking, because different respondents may be using different criteria to evaluate their feelings towards the group.

The authors describe how using anchoring vignettes corrects for these problems by providing respondents with 5 vignettes representing different areas of the 101 degree thermometer scale. Respondents are asked to place each vignette on the feeling thermometer and also rank themselves.

It is suggested that an added value of using anchoring vignettes for feeling thermometers is that socially undesirable feeling will be easier to detect. The use of anchoring vignettes in this study provides preliminary evidence of their effectiveness when examining feelings towards African Americans and homosexuals.

Anchoring Vignettes and the International Comparison of Public Sector Performance

Nigel Rice, Silvana Robone, and Peter C. Smith (from the Centre for Health Economics, University of York) have written a paper on the use of anchoring vignettes to enhance the international comparability of public sector performance.

Using data on health systems responsiveness across 18 OECD countries (contained within the World Health Survey), the authors outline the issues that arise in comparative inference that relies on respondent self-reports. The problem of reporting bias is described and illustrated together with potential solutions brought about through the use of anchoring vignettes. The utility of vignettes to aid cross-country analyses and its implications for comparative inference of health system performance are discussed.

Anchoring Vignettes: Sample Selection Issues and Longitudinal Aspects

Omar Paccagnella from the Dept. of Economics at the University of Padua has a working paper on "Anchoring Vignettes with Sample Selection". The paper aims at extending the standard (hopit) model for estimating vignettes in order to allow the specification of some selection variables.

The concern about sample selection arises from when a respondent in the SHARE (ageing) study completes the main CAPI questionnaire, but does not fill in the extra questions that they have been randomly assigned to - which are anchoring vignettes. Paccagnella states that fitting models to the observed sample ignoring potential selection bias may lead to inconsistent estimates. His findings show that there is evidence of sample selection effects even in the case of high rates of collected vignettes (higher than 85%), but in such cases the bias induced by the selection mechanism is negligible.

Paccagnella also has a working paper in presentation format: on using anchoring vignettes in a longitudinal context, examining work disability reporting from SHARE. This work concludes that when moving from one wave of the SHARE study to the other, that individual thresholds shift upwards and that respondents assess a work limitation less easily in 2006 than in 2004. Also, variations over time in work disability reporting are reported to be much stronger than variations across countries.

Paccagnella has also written on using vignettes to enhance the comparability of self-rated life satisfaction, using the SHARE data. And a paper on voluntary private health insurance for the over-50's, also using the SHARE data.

Time-sharing Experiments for the Social Sciences

Time-sharing Experiments for the Social Sciences (TESS) is an NSF infrastructure project that offers researchers opportunities to test their experimental ideas on large, diverse, randomly-selected subject populations. Investigators submit proposals for experimental studies, and TESS fields selected proposals on a random sample of the United States population using the Internet.

Dan Hopkins and Gary King have successfully submitted an experiment: "Priming to Improve Survey Measurement through Anchoring Vignettes". Their experimental module employed vignettes placed before or after a self-assessment question to determine whether answering vignettes first improved respondents’ capacities to provide meaningful responses. They find that: 1) asking individuals to assess themselves immediately after hearing the vignettes produced responses that were more closely related to key covariates; and 2) inconsistent responses were more likely when individuals were asked to compare themselves directly to hypothetical individuals.

Liam mentioned the relevant paper on the blog before (here). Interestingly, TESS data will be made available to other investigators one year after they are first made available to the authors of successful proposals. The data from the King and Hopkins experiment (and accompanying documentation) is available for download (data in SPSS .sav format).

Geary researchers may also be interseted to know that Howard Schuman (University of Michigan) successfully submitted an experiment entitled: "Recall vs. Judgment: Open-Closed Question Differences in Studying Collective Memory".
Eric Oliver (University of Chicago) and Taeku Lee (UC Berkeley) successfully submitted an experiment entitled "Measuring Perceptions and Attitudes about Overweight and Obesity".

Anchoring Vignettes and the "Rose-Coloured Glasses" Effect in Parents' School Satisfaction

Jack Buckley (Professor of Humanities and Social Sciences at the Steinhardt School, NYU), was mentioned on this blog before in relation to findings he produced on survey context effects in the use of anchoring vignettes. The post is available here.

Buckley has produced other work using anchoring vignettes - related to parents' school satisfaction. He used the chopit model (estimated in GLAMM) to examine the "rose-coloured glasses" effect, which refers to parents reporting higher levels of satisfaction with a school solely or partially as a justification for the effort expended in the choice process. This reminds me of the cognitive dissonance problem, along the lines of "I made the right choice" (when I know I really didn't).

Initially, Buckley finds that parents in charter schools evaluate their child's school more highly and are more satisfied with many dimensions of those schools than parents with children in traditional public schools. (Charter schools are elementary or secondary schools in the United States that receive public money but have been freed from some of the rules, regulations, and statutes that apply to other public schools - more on this here). After applying the anchoring vignettes technique, Buckley shows that parents who change to a charter school are actually tougher graders of the new school.

These results are reported in a book by Buckley and Mark Schneider: "Charter Schools - Hope or Hype?". The publication is currently available courtesy of Google Books here; if you scroll to page 191 you'll find the relevant section. Buckley's vignettes are available to view on Gary King's example page: here, here and here.

Why Would Subjective Measures of Skills-Matching Be Preferred Over Objective Measures?

Before I answer this question, I will remind readers that the idea of "matching" describes the extent of skills-match between Ph.D. training and subsequent employment (It's a different concept to over-education). Using the National Science Foundation's (NSF) `Survey of Doctorate Recipients' (SDR); Bender and Heywood (2006) report that approximately one-sixth of academics in the United States report some degree of mismatch. This mismatch is associated with substantially lower earnings, lower job satisfaction and a higher rate of turnover (Bender and Heywood, 2006). The question on `matching' in the SDR is collected because the (US) National Research Council made a demand for data that shows the extent of integration between "occupational detail and academic training" i.e. `skills-matching'.

The literature on skills-matching is small; there is at least one study using objective data, and a few more using self-reported data. Nordin, Persson and Rooth (2008) is a study that was already mentioned by me on the blog this week (see here). These authors add to the small literature on the consequences of (objective) skills-matching; they use microdata collected by Statistics Sweden, but are forced to drop 36 percent of their sample due to restrictions on fields of education to well-defined categories. The authors state that this approach is necessary because some fields of education (e.g. in the humanities and languages) are either vague or cannot easily be matched with any specific occupation. Also, the authors exclude a further 11 percent of their sample because of missing occupation data.

Robst (2007) discusses other instances where objective measures of skills-matching may be problematic. For example, "many college majors provide students with a broad range of skills... that apply to different occupations. It would be difficult to develop an algorithm for determining whether a major and a job are unrelated... individual assessments, while perhaps subjective, are expected to provide important information." One way around these problems is to use self-rated measures of skill-matching, augmented by the anchoring vignettes technique (see King et al; 2004: here) for enhancing the comparability of survey responses. I mentioned ongoing work on this here last week.

hopkins and king - improving vignettes

a new paper linked below (from the Harvard website) examines some improvements to the practice of vignettes

http://gking.harvard.edu/files/implement.pdf

Optimal Design of Anchoring Vignettes

Jack Buckley (Professor of Humanities and Social Sciences at the Steinhardt School of Culture, Education, and Human Development, NYU), has recently produced some findings on survey context effects in the use of anchoring vignettes. Using data from a randomized survey experiment Buckley investigates whether analyses based on anchoring vignettes may be vulnerable to the introduction of "survey artifacts" due to vignette ordering or the placement of the self-assessment item relative to the vignettes. He finds several patterns of bias due to context effects, and recommends that researchers using anchoring vignettes should consider randomization or other methods to mitigate these problems. Read more about these findings here.

Prior to his role at the Steinhardt School of Culture, Education, and Human Development, Professor Buckley was the Deputy Commissioner of the National Center for Education Statistics, the Federal statistical agency responsible for collecting data in all areas of education in the U.S. Prior to joining NCES, he worked to improve statistical methodology in the federal intelligence community.