  With the rapid growth of crowdsourcing platforms it has become easy and
relatively inexpensive to collect a dataset labeled by multiple annotators in a
short time. However due to the lack of control over the quality of the
annotators, some abnormal annotators may be affected by position bias which can
potentially degrade the quality of the final consensus labels. In this paper we
introduce a statistical framework to model and detect annotator's position bias
in order to control the false discovery rate (FDR) without a prior knowledge on
the amount of biased annotators - the expected fraction of false discoveries
among all discoveries being not too high, in order to assure that most of the
discoveries are indeed true and replicable. The key technical development
relies on some new knockoff filters adapted to our problem and new algorithms
based on the Inverse Scale Space dynamics whose discretization is potentially
suitable for large scale crowdsourcing data analysis. Our studies are supported
by experiments with both simulated examples and real-world data. The proposed
framework provides us a useful tool for quantitatively studying annotator's
abnormal behavior in crowdsourcing data arising from machine learning,
sociology, computer vision, multimedia, etc.
