  Support Vector Machines have been successfully used for one-class
classification (OCSVM, SVDD) when trained on clean data, but they work much
worse on dirty data: outliers present in the training data tend to become
support vectors, and are hence considered "normal". In this article, we improve
the effectiveness to detect outliers in dirty training data with a leave-out
strategy: by temporarily omitting one candidate at a time, this point can be
judged using the remaining data only. We show that this is more effective at
scoring the outlierness of points than using the slack term of existing
SVM-based approaches. Identified outliers can then be removed from the data,
such that outliers hidden by other outliers can be identified, to reduce the
problem of masking. Naively, this approach would require training N individual
SVMs (and training $O(N^2)$ SVMs when iteratively removing the worst outliers
one at a time), which is prohibitively expensive. We will discuss that only
support vectors need to be considered in each step and that by reusing SVM
parameters and weights, this incremental retraining can be accelerated
substantially. By removing candidates in batches, we can further improve the
processing time, although it obviously remains more costly than training a
single SVM.
