  Hierarchies of concepts are useful in many applications from navigation to
organization of objects. Usually, a hierarchy is created in a centralized
manner by employing a group of domain experts, a time-consuming and expensive
process. The experts often design one single hierarchy to best explain the
semantic relationships among the concepts, and ignore the natural uncertainty
that may exist in the process. In this paper, we propose a crowdsourcing system
to build a hierarchy and furthermore capture the underlying uncertainty. Our
system maintains a distribution over possible hierarchies and actively selects
questions to ask using an information gain criterion. We evaluate our
methodology on simulated data and on a set of real world application domains.
Experimental results show that our system is robust to noise, efficient in
picking questions, cost-effective and builds high quality hierarchies.
