The possibility of eliciting the wisdom of crowds has been discussed before on the blog. Prelec described how this might be done in Science in 2004 and produced some results in 2006.
Science 15 October 2004:
Vol. 306. no. 5695, pp. 462 - 466
A Bayesian Truth Serum for Subjective Data
Draen Prelec
Subjective judgments, an essential information source for science and policy, are problematic because there are no public criteria for assessing judgmental truthfulness. I present a scoring method for eliciting truthful subjective data in situations where objective truth is unknowable. The method assigns high scores not to the most common answers but to the answers that are more common than collectively predicted, with predictions drawn from the same population. This simple adjustment in the scoring criterion removes all bias in favor of consensus: Truthful answers maximize expected score even for respondents who believe that their answer represents a minority view.
No comments:
Post a Comment