  Famous for its superior performance, deep learning (DL) has been popularly
used within many applications, which also at the same time attracts various
threats to the models. One primary threat is from adversarial attacks.
Researchers have intensively studied this threat for several years and proposed
dozens of approaches to create adversarial examples (AEs). But most of the
approaches are only evaluated on limited models and datasets (e.g., MNIST,
CIFAR-10). Thus, the effectiveness of attacking real-world DL models is not
quite clear. In this paper, we perform the first systematic study of
adversarial attacks on real-world DNN models and provide a real-world model
dataset named RWM. Particularly, we design a suite of approaches to adapt
current AE generation algorithms to the diverse real-world DL models, including
automatically extracting DL models from Android apps, capturing the inputs and
outputs of the DL models in apps, generating AEs and validating them by
observing the apps' execution. For black-box DL models, we design a
semantic-based approach to build suitable datasets and use them for training
substitute models when performing transfer-based attacks. After analyzing 245
DL models collected from 62,583 real-world apps, we have a unique opportunity
to understand the gap between real-world DL models and contemporary AE
generation algorithms. To our surprise, the current AE generation algorithms
can only directly attack 6.53% of the models. Benefiting from our approach, the
success rate upgrades to 47.35%.
