  We study the problem of organizing a collection of objects - images, videos -
into clusters, using crowdsourcing. This problem is notoriously hard for
computers to do automatically, and even with crowd workers, is challenging to
orchestrate: (a) workers may cluster based on different latent hierarchies or
perspectives; (b) workers may cluster at different granularities even when
clustering using the same perspective; and (c) workers may only see a small
portion of the objects when deciding how to cluster them (and therefore have
limited understanding of the "big picture"). We develop cost-efficient,
accurate algorithms for identifying the consensus organization (i.e., the
organizing perspective most workers prefer to employ), and incorporate these
algorithms into a cost-effective workflow for organizing a collection of
objects, termed ORCHESTRA. We compare our algorithms with other algorithms for
clustering, on a variety of real-world datasets, and demonstrate that ORCHESTRA
organizes items better and at significantly lower costs.
