  In this work, we propose several enhancements to a geometric transformation
based model for anomaly detection in images (GeoTranform). The model assumes
that the anomaly class is unknown and that only inlier samples are available
for training. We introduce new filter based transformations useful for
detecting anomalies in astronomical images, that highlight artifact properties
to make them more easily distinguishable from real objects. In addition, we
propose a transformation selection strategy that allows us to find
indistinguishable pairs of transformations. This results in an improvement of
the area under the Receiver Operating Characteristic curve (AUROC) and accuracy
performance, as well as in a dimensionality reduction. The models were tested
on astronomical images from the High Cadence Transient Survey (HiTS) and Zwicky
Transient Facility (ZTF) datasets. The best models obtained an average AUROC of
99.20% for HiTS and 91.39% for ZTF. The improvement over the original
GeoTransform algorithm and baseline methods such as One-Class Support Vector
Machine, and deep learning based methods is significant both statistically and
in practice.
