  Machine learning, with its myriad applications, has become an integral
component of numerous technological systems. A common practice in this domain
is the use of transfer learning, where a pre-trained model's architecture,
readily available to the public, is fine-tuned to suit specific tasks. As
Machine Learning as a Service (MLaaS) platforms increasingly use pre-trained
models in their backends, it's crucial to safeguard these architectures and
understand their vulnerabilities. In this work, we present an approach based on
the observation that the classification patterns of adversarial images can be
used as a means to steal the models. Furthermore, the adversarial image
classifications in conjunction with timing side channels can lead to a model
stealing method. Our approach, designed for typical user-level access in remote
MLaaS environments exploits varying misclassifications of adversarial images
across different models to fingerprint several renowned Convolutional Neural
Network (CNN) and Vision Transformer (ViT) architectures. We utilize the
profiling of remote model inference times to reduce the necessary adversarial
images, subsequently decreasing the number of queries required. We have
presented our results over 27 pre-trained models of different CNN and ViT
architectures using CIFAR-10 dataset and demonstrate a high accuracy of 88.8%
while keeping the query budget under 20.
