  Current deep learning methods for anomaly detection in text rely on
supervisory signals in inliers that may be unobtainable or bespoke
architectures that are difficult to tune. We study a simpler alternative:
fine-tuning Transformers on the inlier data with self-supervised objectives and
using the losses as an anomaly score. Overall, the self-supervision approach
outperforms other methods under various anomaly detection scenarios, improving
the AUROC score on semantic anomalies by 11.6% and on syntactic anomalies by
22.8% on average. Additionally, the optimal objective and resultant learnt
representation depend on the type of downstream anomaly. The separability of
anomalies and inliers signals that a representation is more effective for
detecting semantic anomalies, whilst the presence of narrow feature directions
signals a representation that is effective for detecting syntactic anomalies.
