  Anomaly detection-based spoof attack detection is a recent development in
face Presentation Attack Detection (fPAD), where a spoof detector is learned
using only non-attacked images of users. These detectors are of practical
importance as they are shown to generalize well to new attack types. In this
paper, we present a deep-learning solution for anomaly detection-based spoof
attack detection where both classifier and feature representations are learned
together end-to-end. First, we introduce a pseudo-negative class during
training in the absence of attacked images. The pseudo-negative class is
modeled using a Gaussian distribution whose mean is calculated by a weighted
running mean. Secondly, we use pairwise confusion loss to further regularize
the training process. The proposed approach benefits from the representation
learning power of the CNNs and learns better features for fPAD task as shown in
our ablation study. We perform extensive experiments on four publicly available
datasets: Replay-Attack, Rose-Youtu, OULU-NPU and Spoof in Wild to show the
effectiveness of the proposed approach over the previous methods. Code is
available at: \url{https://github.com/yashasvi97/IJCB2020_anomaly}
