  Pilot studies are an essential cornerstone of the design of crowdsourcing
campaigns, yet they are often only mentioned in passing in the scholarly
literature. A lack of details surrounding pilot studies in crowdsourcing
research hinders the replication of studies and the reproduction of findings,
stalling potential scientific advances. We conducted a systematic literature
review on the current state of pilot study reporting at the intersection of
crowdsourcing and HCI research. Our review of ten years of literature included
171 articles published in the proceedings of the Conference on Human
Computation and Crowdsourcing (AAAI HCOMP) and the ACM Digital Library. We
found that pilot studies in crowdsourcing research (i.e., crowd pilot studies)
are often under-reported in the literature. Important details, such as the
number of workers and rewards to workers, are often not reported. On the basis
of our findings, we reflect on the current state of practice and formulate a
set of best practice guidelines for reporting crowd pilot studies in
crowdsourcing research. We also provide implications for the design of
crowdsourcing platforms and make practical suggestions for supporting crowd
pilot study reporting.
