  In this paper, we propose dictionary attacks against speaker verification - a
novel attack vector that aims to match a large fraction of speaker population
by chance. We introduce a generic formulation of the attack that can be used
with various speech representations and threat models. The attacker uses
adversarial optimization to maximize raw similarity of speaker embeddings
between a seed speech sample and a proxy population. The resulting master voice
successfully matches a non-trivial fraction of people in an unknown population.
Adversarial waveforms obtained with our approach can match on average 69% of
females and 38% of males enrolled in the target system at a strict decision
threshold calibrated to yield false alarm rate of 1%. By using the attack with
a black-box voice cloning system, we obtain master voices that are effective in
the most challenging conditions and transferable between speaker encoders. We
also show that, combined with multiple attempts, this attack opens even more to
serious issues on the security of these systems.
