  Modern, state-of-the-art deep learning approaches yield human like
performance in numerous object detection and classification tasks. The
foundation for their success is the availability of training datasets of
substantially high quantity, which are expensive to create, especially in the
field of medical imaging. Recently, crowdsourcing has been applied to create
large datasets for a broad range of disciplines. This study aims to explore the
challenges and opportunities of crowd-algorithm collaboration for the object
detection task of grading cytology whole slide images. We compared the
classical crowdsourcing performance of twenty participants with their results
from crowd-algorithm collaboration. All participants performed both modes in
random order on the same twenty images. Additionally, we introduced artificial
systematic flaws into the precomputed annotations to estimate a bias towards
accepting precomputed annotations. We gathered 9524 annotations on 800 images
from twenty participants organised into four groups in concordance to their
level of expertise with cytology. The crowd-algorithm mode improved on average
the participants' classification accuracy by 7%, the mean average precision by
8% and the inter-observer Fleiss' kappa score by 20%, and reduced the time
spent by 31%. However, two thirds of the artificially modified false labels
were not recognised as such by the contributors. This study shows that
crowd-algorithm collaboration is a promising new approach to generate large
datasets when it is ensured that a carefully designed setup eliminates
potential biases.
