Characterization of experts in crowdsourcing platforms
  Crowdsourcing platforms enable to propose simple human intelligence tasks to
a large number of participants who realise these tasks. The workers often
receive a small amount of money or the platforms include some other incentive
mechanisms, for example they can increase the workers reputation score, if they
complete the tasks correctly. We address the problem of identifying experts
among participants, that is, workers, who tend to answer the questions
correctly. Knowing who are the reliable workers could improve the quality of
knowledge one can extract from responses. As opposed to other works in the
literature, we assume that participants can give partial or incomplete
responses, in case they are not sure that their answers are correct. We model
such partial or incomplete responses with the help of belief functions, and we
derive a measure that characterizes the expertise level of each participant.
This measure is based on precise and exactitude degrees that represent two
parts of the expertise level. The precision degree reflects the reliability
level of the participants and the exactitude degree reflects the knowledge
level of the participants. We also analyze our model through simulation and
demonstrate that our richer model can lead to more reliable identification of
experts.
