  Given a (machine learning) classifier and a collection of unlabeled data, how
can we efficiently identify misclassification patterns presented in this
dataset? To address this problem, we propose a human-machine collaborative
framework that consists of a team of human annotators and a sequential
recommendation algorithm. The recommendation algorithm is conceptualized as a
stochastic sampler that, in each round, queries the annotators a subset of
samples for their true labels and obtains the feedback information on whether
the samples are misclassified. The sampling mechanism needs to balance between
discovering new patterns of misclassification (exploration) and confirming the
potential patterns of classification (exploitation). We construct a
determinantal point process, whose intensity balances the
exploration-exploitation trade-off through the weighted update of the posterior
at each round to form the generator of the stochastic sampler. The numerical
results empirically demonstrate the competitive performance of our framework on
multiple datasets at various signal-to-noise ratios.
