  In this paper, we explore how to efficiently combine crowdsourcing and
machine intelligence for the problem of document screening, where we need to
screen documents with a set of machine-learning filters. Specifically, we focus
on building a set of machine learning classifiers that evaluate documents, and
then screen them efficiently. It is a challenging task since the budget is
limited and there are countless number of ways to spend the given budget on the
problem. We propose a multi-label active learning screening specific sampling
technique -- objective-aware sampling -- for querying unlabelled documents for
annotating. Our algorithm takes a decision on which machine filter need more
training data and how to choose unlabeled items to annotate in order to
minimize the risk of overall classification errors rather than minimizing a
single filter error. We demonstrate that objective-aware sampling significantly
outperforms the state of the art active learning sampling strategies.
