  How should one combine noisy information from diverse sources to make an
inference about an objective ground truth? This frequently recurring, normative
question lies at the core of statistics, machine learning, policy-making, and
everyday life. It has been called "combining forecasts", "meta-analysis",
"ensembling", and the "MLE approach to voting", among other names. Past studies
typically assume that noisy votes are identically and independently distributed
(i.i.d.), but this assumption is often unrealistic. Instead, we assume that
votes are independent but not necessarily identically distributed and that our
ensembling algorithm has access to certain auxiliary information related to the
underlying model governing the noise in each vote. In our present work, we: (1)
define our problem and argue that it reflects common and socially relevant real
world scenarios, (2) propose a multi-arm bandit noise model and count-based
auxiliary information set, (3) derive maximum likelihood aggregation rules for
ranked and cardinal votes under our noise model, (4) propose, alternatively, to
learn an aggregation rule using an order-invariant neural network, and (5)
empirically compare our rules to common voting rules and naive
experience-weighted modifications. We find that our rules successfully use
auxiliary information to outperform the naive baselines.
