  We aim for image-based novelty detection. Despite considerable progress,
existing models either fail or face a dramatic drop under the so-called
"near-distribution" setting, where the differences between normal and anomalous
samples are subtle. We first demonstrate existing methods experience up to 20%
decrease in performance in the near-distribution setting. Next, we propose to
exploit a score-based generative model to produce synthetic near-distribution
anomalous data. Our model is then fine-tuned to distinguish such data from the
normal samples. We provide a quantitative as well as qualitative evaluation of
this strategy, and compare the results with a variety of GAN-based models.
Effectiveness of our method for both the near-distribution and standard novelty
detection is assessed through extensive experiments on datasets in diverse
applications such as medical images, object classification, and quality
control. This reveals that our method considerably improves over existing
models, and consistently decreases the gap between the near-distribution and
standard novelty detection performance. The code repository is available at
https://github.com/rohban-lab/FITYMI.
