  Crowdsourcing has been widely used to efficiently obtain labeled datasets for
supervised learning from large numbers of human resources at low cost. However,
one of the technical challenges in obtaining high-quality results from
crowdsourcing is dealing with the variability and bias caused by the fact that
it is humans execute the work, and various studies have addressed this issue to
improve the quality by integrating redundantly collected responses. In this
study, we focus on the observation bias in crowdsourcing. Variations in the
frequency of worker responses and the complexity of tasks occur, which may
affect the aggregation results when they are correlated with the quality of the
responses. We also propose statistical aggregation methods for crowdsourcing
responses that are combined with an observational data bias removal method used
in causal inference. Through experiments using both synthetic and real datasets
with/without artificially injected spam and colluding workers, we verify that
the proposed method improves the aggregation accuracy in the presence of strong
observation biases and robustness to both spam and colluding workers.
