  Current text-to-speech algorithms produce realistic fakes of human voices,
making deepfake detection a much-needed area of research. While researchers
have presented various techniques for detecting audio spoofs, it is often
unclear exactly why these architectures are successful: Preprocessing steps,
hyperparameter settings, and the degree of fine-tuning are not consistent
across related work. Which factors contribute to success, and which are
accidental? In this work, we address this problem: We systematize audio
spoofing detection by re-implementing and uniformly evaluating architectures
from related work. We identify overarching features for successful audio
deepfake detection, such as using cqtspec or logspec features instead of
melspec features, which improves performance by 37% EER on average, all other
factors constant. Additionally, we evaluate generalization capabilities: We
collect and publish a new dataset consisting of 37.9 hours of found audio
recordings of celebrities and politicians, of which 17.2 hours are deepfakes.
We find that related work performs poorly on such real-world data (performance
degradation of up to one thousand percent). This may suggest that the community
has tailored its solutions too closely to the prevailing ASVSpoof benchmark and
that deepfakes are much harder to detect outside the lab than previously
thought.
