  Wisdom of the crowd revealed a striking fact that the majority answer from a
crowd is often more accurate than any individual expert. We observed the same
story in machine learning--ensemble methods leverage this idea to combine
multiple learning algorithms to obtain better classification performance. Among
many popular examples is the celebrated Random Forest, which applies the
majority voting rule in aggregating different decision trees to make the final
prediction. Nonetheless, these aggregation rules would fail when the majority
is more likely to be wrong. In this paper, we extend the idea proposed in
Bayesian Truth Serum that "a surprisingly more popular answer is more likely
the true answer" to classification problems. The challenge for us is to define
or detect when an answer should be considered as being "surprising". We present
two machine learning aided methods which aim to reveal the truth when it is
minority instead of majority who has the true answer. Our experiments over
real-world datasets show that better classification performance can be obtained
compared to always trusting the majority voting. Our proposed methods also
outperform popular ensemble algorithms. Our approach can be generically applied
as a subroutine in ensemble methods to replace majority voting rule.
