  People are not always receptive to their voice data being collected and
misused. Training the audio intelligence systems needs these data to build
useful features, but the cost for getting permissions or purchasing data is
very high, which inevitably encourages hackers to collect these voice data
without people's awareness. To discourage the hackers from proactively
collecting people's voice data, we are the first to propose a clean-label
poisoning attack, called WaveFuzz, which can prevent intelligence audio models
from building useful features from protected (poisoned) voice data but still
preserve the semantic information to the humans. Specifically, WaveFuzz
perturbs the voice data to cause Mel Frequency Cepstral Coefficients (MFCC)
(typical representations of audio signals) to generate the poisoned frequency
features. These poisoned features are then fed to audio prediction models,
which degrades the performance of audio intelligence systems. Empirically, we
show the efficacy of WaveFuzz by attacking two representative types of
intelligent audio systems, i.e., speaker recognition system (SR) and speech
command recognition system (SCR). For example, the accuracies of models are
declined by $19.78\%$ when only $10\%$ of the poisoned voice data is to
fine-tune models, and the accuracies of models declined by $6.07\%$ when only
$10\%$ of the training voice data is poisoned. Consequently, WaveFuzz is an
effective technique that enables people to fight back to protect their own
voice data, which sheds new light on ameliorating privacy issues.
