  In heterogeneous rank aggregation problems, users often exhibit various
accuracy levels when comparing pairs of items. Thus a uniform querying strategy
over users may not be optimal. To address this issue, we propose an
elimination-based active sampling strategy, which estimates the ranking of
items via noisy pairwise comparisons from users and improves the users' average
accuracy by maintaining an active set of users. We prove that our algorithm can
return the true ranking of items with high probability. We also provide a
sample complexity bound for the proposed algorithm which is better than that of
non-active strategies in the literature. Experiments are provided to show the
empirical advantage of the proposed methods over the state-of-the-art
baselines.
