  Motivated by the common strategic activities in crowdsourcing labeling, we
study the problem of sequential eliciting information without verification
(EIWV) for workers with a heterogeneous and unknown crowd. We propose a
reinforcement learning-based approach that is effective against a wide range of
settings including potential irrationality and collusion among workers. With
the aid of a costly oracle and the inference method, our approach dynamically
decides the oracle calls and gains robustness even under the presence of
frequent collusion activities. Extensive experiments show the advantage of our
approach. Our results also present the first comprehensive experiments of EIWV
on large-scale real datasets and the first thorough study of the effects of
environmental variables.
