  This paper reports on the challenges and lessons we learned while running
controlled experiments in crowdsourcing platforms. Crowdsourcing is becoming an
attractive technique to engage a diverse and large pool of subjects in
experimental research, allowing researchers to achieve levels of scale and
completion times that would otherwise not be feasible in lab settings. However,
the scale and flexibility comes at the cost of multiple and sometimes unknown
sources of bias and confounding factors that arise from technical limitations
of crowdsourcing platforms and from the challenges of running controlled
experiments in the "wild". In this paper, we take our experience in running
systematic evaluations of task design as a motivating example to explore,
describe, and quantify the potential impact of running uncontrolled
crowdsourcing experiments and derive possible coping strategies. Among the
challenges identified, we can mention sampling bias, controlling the assignment
of subjects to experimental conditions, learning effects, and reliability of
crowdsourcing results. According to our empirical studies, the impact of
potential biases and confounding factors can amount to a 38\% loss in the
utility of the data collected in uncontrolled settings; and it can
significantly change the outcome of experiments. These issues ultimately
inspired us to implement CrowdHub, a system that sits on top of major
crowdsourcing platforms and allows researchers and practitioners to run
controlled crowdsourcing projects.
