  Outlier detection (OD) literature exhibits numerous algorithms as it applies
to diverse domains. However, given a new detection task, it is unclear how to
choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in
unsupervised settings. HP tuning is an ever-growing problem with the arrival of
many new detectors based on deep learning, which usually come with a long list
of HPs. Surprisingly, the issue of model selection in the outlier mining
literature has been "the elephant in the room"; a significant factor in
unlocking the utmost potential of deep methods, yet little said or done to
systematically tackle the issue. In the first part of this paper, we conduct
the first large-scale analysis on the HP sensitivity of deep OD methods, and
through more than 35,000 trained models, quantitatively demonstrate that model
selection is inevitable. Next, we design a HP-robust and scalable deep
hyper-ensemble model called ROBOD that assembles models with varying HP
configurations, bypassing the choice paralysis. Importantly, we introduce novel
strategies to speed up ensemble training, such as parameter sharing,
batch/simultaneous training, and data subsampling, that allow us to train fewer
models with fewer parameters. Extensive experiments on both image and tabular
datasets show that ROBOD achieves and retains robust, state-of-the-art
detection performance as compared to its modern counterparts, while taking only
$2$-$10$\% of the time by the naive hyper-ensemble with independent training.
