  Outlier detection is the identification of points in a dataset that do not
conform to the norm. Outlier detection is highly sensitive to the choice of the
detection algorithm and the feature subspace used by the algorithm. Extracting
domain-relevant insights from outliers needs systematic exploration of these
choices since diverse outlier sets could lead to complementary insights. This
challenge is especially acute in an interactive setting, where the choices must
be explored in a time-constrained manner. In this work, we present REMIX, the
first system to address the problem of outlier detection in an interactive
setting. REMIX uses a novel mixed integer programming (MIP) formulation for
automatically selecting and executing a diverse set of outlier detectors within
a time limit. This formulation incorporates multiple aspects such as (i) an
upper limit on the total execution time of detectors (ii) diversity in the
space of algorithms and features, and (iii) meta-learning for evaluating the
cost and utility of detectors. REMIX provides two distinct ways for the analyst
to consume its results: (i) a partitioning of the detectors explored by REMIX
into perspectives through low-rank non-negative matrix factorization; each
perspective can be easily visualized as an intuitive heatmap of experiments
versus outliers, and (ii) an ensembled set of outliers which combines outlier
scores from all detectors. We demonstrate the benefits of REMIX through
extensive empirical validation on real-world data.
