  This paper discusses how crowd and machine classifiers can be efficiently
combined to screen items that satisfy a set of predicates. We show that this is
a recurring problem in many domains, present machine-human (hybrid) algorithms
that screen items efficiently and estimate the gain over human-only or
machine-only screening in terms of performance and cost. We further show how,
given a new classification problem and a set of classifiers of unknown accuracy
for the problem at hand, we can identify how to manage the cost-accuracy trade
off by progressively determining if we should spend budget to obtain test data
(to assess the accuracy of the given classifiers), or to train an ensemble of
classifiers, or whether we should leverage the existing machine classifiers
with the crowd, and in this case how to efficiently combine them based on their
estimated characteristics to obtain the classification. We demonstrate that the
techniques we propose obtain significant cost/accuracy improvements with
respect to the leading classification algorithms.
