  Deep neural networks (DNNs) have achieved great success in various
applications due to their strong expressive power. However, recent studies have
shown that DNNs are vulnerable to adversarial examples which are manipulated
instances targeting to mislead DNNs to make incorrect predictions. Currently,
most such adversarial examples try to guarantee "subtle perturbation" by
limiting the $L_p$ norm of the perturbation. In this paper, we aim to explore
the impact of semantic manipulation on DNNs predictions by manipulating the
semantic attributes of images and generate "unrestricted adversarial examples".
  In particular, we propose an algorithm \emph{SemanticAdv} which leverages
disentangled semantic factors to generate adversarial perturbation by altering
controlled semantic attributes to fool the learner towards various
"adversarial" targets. We conduct extensive experiments to show that the
semantic based adversarial examples can not only fool different learning tasks
such as face verification and landmark detection, but also achieve high
targeted attack success rate against \emph{real-world black-box} services such
as Azure face verification service based on transferability.
  To further demonstrate the applicability of \emph{SemanticAdv} beyond face
recognition domain, we also generate semantic perturbations on street-view
images. Such adversarial examples with controlled semantic manipulation can
shed light on further understanding about vulnerabilities of DNNs as well as
potential defensive approaches.
