  Anomaly detectors are often designed to catch statistical anomalies.
End-users typically do not have interest in all of the detected outliers, but
only those relevant to their application. Given an existing black-box
sequential anomaly detector, this paper proposes a method to improve its user
relevancy using a small number of human feedback. As our first contribution,
the method is agnostic to the detector: it only assumes access to its anomaly
scores, without requirement on any additional information inside it. Inspired
by a fact that anomalies are of different types, our approach identifies these
types and utilizes user feedback to assign relevancy to types. This relevancy
score, as our second contribution, is used to adjust the subsequent anomaly
selection process. Empirical results on synthetic and real-world datasets show
that our approach yields significant improvements on precision and recall over
a range of anomaly detectors.
