  We consider the problem of recovering the ground truth ordering (ranking,
top-$k$, or others) over a large number of alternatives. The wisdom of crowd is
a heuristic approach based on Condorcet's Jury theorem to address this problem
through collective opinions. This approach fails to recover the ground truth
when the majority of the crowd is misinformed. The surprisingly popular (SP)
algorithm cite{prelec2017solution} is an alternative approach that is able to
recover the ground truth even when experts are in minority. The SP algorithm
requires the voters to predict other voters' report in the form of a full
probability distribution over all rankings of alternatives. However, when the
number of alternatives, $m$, is large, eliciting the prediction report or even
the vote over $m$ alternatives might be too costly. In this paper, we design a
scalable alternative of the SP algorithm which only requires eliciting partial
preferences from the voters, and propose new variants of the SP algorithm. In
particular, we propose two versions -- Aggregated-SP and Partial-SP -- that ask
voters to report vote and prediction on a subset of size $k$ ($\ll m$) in terms
of top alternative, partial rank, or an approval set. Through a large-scale
crowdsourcing experiment on MTurk, we show that both of our approaches
outperform conventional preference aggregation algorithms for the recovery of
ground truth rankings, when measured in terms of Kendall-Tau distance and
Spearman's $\rho$. We further analyze the collected data and demonstrate that
voters' behavior in the experiment, including the minority of the experts, and
the SP phenomenon, can be correctly simulated by a concentric mixtures of
Mallows model. Finally, we provide theoretical bounds on the sample complexity
of SP algorithms with partial rankings to demonstrate the theoretical
guarantees of the proposed methods.
