  Automatic speaker verification (ASV) systems in practice are greatly
vulnerable to spoofing attacks. The latest voice conversion technologies are
able to produce perceptually natural sounding speech that mimics any target
speakers. However, the perceptual closeness to a speaker's identity may not be
enough to deceive an ASV system. In this work, we propose a framework that uses
the output scores of an ASV system as the feedback to a voice conversion
system. The attacker framework is a black-box adversary that steals one's voice
identity, because it does not require any knowledge about the ASV system but
the system outputs. Experimental results conducted on ASVspoof 2019 database
confirm that the proposed feedback-controlled voice conversion framework
produces adversarial samples that are more deceptive than the straightforward
voice conversion, thereby boosting the impostor ASV scores. Further, the
perceptual evaluation studies reveal that converted speech does not adversely
affect the voice quality from the baseline system.
