  Speech and speaker recognition systems are employed in a variety of
applications, from personal assistants to telephony surveillance and biometric
authentication. The wide deployment of these systems has been made possible by
the improved accuracy in neural networks. Like other systems based on neural
networks, recent research has demonstrated that speech and speaker recognition
systems are vulnerable to attacks using manipulated inputs. However, as we
demonstrate in this paper, the end-to-end architecture of speech and speaker
systems and the nature of their inputs make attacks and defenses against them
substantially different than those in the image space. We demonstrate this
first by systematizing existing research in this space and providing a taxonomy
through which the community can evaluate future work. We then demonstrate
experimentally that attacks against these models almost universally fail to
transfer. In so doing, we argue that substantial additional work is required to
provide adequate mitigations in this space.
