  When we use the wisdom of the crowds, we usually rank the answers according
to their popularity, especially when we cannot verify the answers. However,
this can be very dangerous when the majority make systematic mistakes. A
fundamental question arises: can we build a hierarchy among the answers
\textit{without any prior} where the higher-ranking answers, which may not be
supported by the majority, are from more sophisticated people? To address the
question, we propose 1) a novel model to describe people's thinking hierarchy;
2) two algorithms to learn the thinking hierarchy without any prior; 3) a novel
open-response based crowdsourcing approach based on the above theoretic
framework. In addition to theoretic justifications, we conduct four empirical
crowdsourcing studies and show that a) the accuracy of the top-ranking answers
learned by our approach is much higher than that of plurality voting (In one
question, the plurality answer is supported by 74 respondents but the correct
answer is only supported by 3 respondents. Our approach ranks the correct
answer the highest without any prior); b) our model has a high goodness-of-fit,
especially for the questions where our top-ranking answer is correct. To the
best of our knowledge, we are the first to propose a thinking hierarchy model
with empirical validations in the general problem-solving scenarios; and the
first to propose a practical open-response based crowdsourcing approach that
beats plurality voting without any prior.
