  Generating adversarial examples is a critical step for evaluating and
improving the robustness of learning machines. So far, most existing methods
only work for classification and are not designed to alter the true performance
measure of the problem at hand. We introduce a novel flexible approach named
Houdini for generating adversarial examples specifically tailored for the final
performance measure of the task considered, be it combinatorial and
non-decomposable. We successfully apply Houdini to a range of applications such
as speech recognition, pose estimation and semantic segmentation. In all cases,
the attacks based on Houdini achieve higher success rate than those based on
the traditional surrogates used to train the models while using a less
perceptible adversarial perturbation.
