  In threshold-based anomaly detection, we want to tune the threshold of a
detector to achieve an acceptable false alarm rate. However, tuning the
threshold is often a non-trivial task due to unknown detector output
distributions. A detector threshold that provides an acceptable false alarm
rate is equivalent to a specific quantile of the detector output distribution.
Therefore, we use quantile estimators based on order statistics to estimate the
detector threshold. The estimation of quantiles from sample data has a more
than a century long tradition and we provide three different distribution-free
finite sample guarantees for a class of quantile estimators. The first is based
on the Dworetzky-Kiefer-Wolfowitz inequality, the second utilizes the
Vysochanskij-Petunin inequality, and the third is based on exact confidence
intervals for a beta distribution. These guarantees are then compared and used
in the detector threshold tuning problem. We use both simulated data as well as
data obtained from an experimental setup with the Temperature Control Lab to
validate the guarantees provided.
