  Given a pre-trained classifier and multiple human experts, we investigate the
task of online classification where model predictions are provided for free but
querying humans incurs a cost. In this practical but under-explored setting,
oracle ground truth is not available. Instead, the prediction target is defined
as the consensus vote of all experts. Given that querying full consensus can be
costly, we propose a general framework for online Bayesian consensus
estimation, leveraging properties of the multivariate hypergeometric
distribution. Based on this framework, we propose a family of methods that
dynamically estimate expert consensus from partial feedback by producing a
posterior over expert and model beliefs. Analyzing this posterior induces an
interpretable trade-off between querying cost and classification performance.
We demonstrate the efficacy of our framework against a variety of baselines on
CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.
