  Automatic speaker verification systems are vulnerable to audio replay attacks
which bypass security by replaying recordings of authorized speakers. Replay
attack detection (RA) detection systems built upon Residual Neural Networks
(ResNet)s have yielded astonishing results on the public benchmark ASVspoof
2019 Physical Access challenge. With most teams using fine-tuned feature
extraction pipelines and model architectures, the generalizability of such
systems remains questionable though. In this work, we analyse the effect of
discriminative feature learning in a multi-task learning (MTL) setting can have
on the generalizability and discriminability of RA detection systems. We use a
popular ResNet architecture optimized by the cross-entropy criterion as our
baseline and compare it to the same architecture optimized by MTL using Siamese
Neural Networks (SNN). It can be shown that SNN outperform the baseline by
relative 26.8 % Equal Error Rate (EER). We further enhance the model's
architecture and demonstrate that SNN with additional reconstruction loss yield
another significant improvement of relative 13.8 % EER.
