  Being able to infer ground truth from the responses of multiple imperfect
advisors is a problem of crucial importance in many decision-making
applications, such as lending, trading, investment, and crowd-sourcing. In
practice, however, gathering answers from a set of advisors has a cost.
Therefore, finding an advisor selection strategy that retrieves a reliable
answer and maximizes the overall utility is a challenging problem. To address
this problem, we propose a novel strategy for optimally selecting a set of
advisers in a sequential binary decision-making setting, where multiple
decisions need to be made over time. Crucially, we assume no access to ground
truth and no prior knowledge about the reliability of advisers. Specifically,
our approach considers how to simultaneously (1) select advisors by balancing
the advisors' costs and the value of making correct decisions, (2) learn the
trustworthiness of advisers dynamically without prior information by asking
multiple advisers, and (3) make optimal decisions without access to the ground
truth, improving this over time. We evaluate our algorithm through several
numerical experiments. The results show that our approach outperforms two other
methods that combine state-of-the-art models.
