  We study the problem of eliciting and aggregating probabilistic information
from multiple agents. In order to successfully aggregate the predictions of
agents, the principal needs to elicit some notion of confidence from agents,
capturing how much experience or knowledge led to their predictions. To
formalize this, we consider a principal who wishes to elicit predictions about
a random variable from a group of Bayesian agents, each of whom have privately
observed some independent samples of the random variable, and hopes to
aggregate the predictions as if she had directly observed the samples of all
agents. Leveraging techniques from Bayesian statistics, we represent confidence
as the number of samples an agent has observed, which is quantified by a
hyperparameter from a conjugate family of prior distributions. This then allows
us to show that if the principal has access to a few samples, she can achieve
her aggregation goal by eliciting predictions from agents using proper scoring
rules. In particular, if she has access to one sample, she can successfully
aggregate the agents' predictions if and only if every posterior predictive
distribution corresponds to a unique value of the hyperparameter. Furthermore,
this uniqueness holds for many common distributions of interest. When this
uniqueness property does not hold, we construct a novel and intuitive mechanism
where a principal with two samples can elicit and optimally aggregate the
agents' predictions.
