  In the face of a new era of generative models, the detection of artificially
generated content has become a matter of utmost importance. The ability to
create credible minute-long music deepfakes in a few seconds on user-friendly
platforms poses a real threat of fraud on streaming services and unfair
competition to human artists. This paper demonstrates the possibility (and
surprising ease) of training classifiers on datasets comprising real audio and
fake reconstructions, achieving a convincing accuracy of 99.8%. To our
knowledge, this marks the first publication of a music deepfake detector, a
tool that will help in the regulation of music forgery. Nevertheless, informed
by decades of literature on forgery detection in other fields, we stress that a
good test score is not the end of the story. We step back from the
straightforward ML framework and expose many facets that could be problematic
with such a deployed detector: calibration, robustness to audio manipulation,
generalisation to unseen models, interpretability and possibility for recourse.
This second part acts as a position for future research steps in the field and
a caveat to a flourishing market of fake content checkers.
