  Data labeling is a necessary but often slow process that impedes the
development of interactive systems for modern data analysis. Despite rising
demand for manual data labeling, there is a surprising lack of work addressing
its high and unpredictable latency. In this paper, we introduce CLAMShell, a
system that speeds up crowds in order to achieve consistently low-latency data
labeling. We offer a taxonomy of the sources of labeling latency and study
several large crowd-sourced labeling deployments to understand their empirical
latency profiles. Driven by these insights, we comprehensively tackle each
source of latency, both by developing novel techniques such as straggler
mitigation and pool maintenance and by optimizing existing methods such as
crowd retainer pools and active learning. We evaluate CLAMShell in simulation
and on live workers on Amazon's Mechanical Turk, demonstrating that our
techniques can provide an order of magnitude speedup and variance reduction
over existing crowdsourced labeling strategies.
