  The Automatic Speaker Verification Spoofing and Countermeasures Challenges
motivate research in protecting speech biometric systems against a variety of
different access attacks. The 2017 edition focused on replay spoofing attacks,
and involved participants building and training systems on a provided dataset
(ASVspoof 2017). More than 60 research papers have so far been published with
this dataset, but none have sought to answer why countermeasures appear
successful in detecting spoofing attacks. This article shows how artefacts
inherent to the dataset may be contributing to the apparent success of
published systems. We first inspect the ASVspoof 2017 dataset and summarize
various artefacts present in the dataset. Second, we demonstrate how
countermeasure models can exploit these artefacts to appear successful in this
dataset. Third, for reliable and robust performance estimates on this dataset
we propose discarding nonspeech segments and silence before and after the
speech utterance during training and inference. We create speech start and
endpoint annotations in the dataset and demonstrate how using them helps
countermeasure models become less vulnerable from being manipulated using
artefacts found in the dataset. Finally, we provide several new benchmark
results for both frame-level and utterance-level models that can serve as new
baselines on this dataset.
