  Anomaly detection attempts at finding examples that deviate from the expected
behaviour. Usually, anomaly detection is tackled from an unsupervised
perspective because anomalous labels are rare and difficult to acquire.
However, the lack of labels makes the anomaly detector have high uncertainty in
some regions, which usually results in poor predictive performance or low user
trust in the predictions. One can reduce such uncertainty by collecting
specific labels using Active Learning (AL), which targets examples close to the
detector's decision boundary. Alternatively, one can increase the user trust by
allowing the detector to abstain from making highly uncertain predictions,
which is called Learning to Reject (LR). One way to do this is by thresholding
the detector's uncertainty based on where its performance is low, which
requires labels to be evaluated. Although both AL and LR need labels, they work
with different types of labels: AL seeks strategic labels, which are evidently
biased, while LR requires i.i.d. labels to evaluate the detector's performance
and set the rejection threshold. Because one usually has a unique label budget,
deciding how to optimally allocate it is challenging. In this paper, we propose
a mixed strategy that, given a budget of labels, decides in multiple rounds
whether to use the budget to collect AL labels or LR labels. The strategy is
based on a reward function that measures the expected gain when allocating the
budget to either side. We evaluate our strategy on 18 benchmark datasets and
compare it to some baselines.
