  With the increased usage of AI accelerators on mobile and edge devices,
on-device machine learning (ML) is gaining popularity. Thousands of proprietary
ML models are being deployed today on billions of untrusted devices. This
raises serious security concerns about model privacy. However, protecting model
privacy without losing access to the untrusted AI accelerators is a challenging
problem. In this paper, we present a novel on-device model inference system,
ShadowNet. ShadowNet protects the model privacy with Trusted Execution
Environment (TEE) while securely outsourcing the heavy linear layers of the
model to the untrusted hardware accelerators. ShadowNet achieves this by
transforming the weights of the linear layers before outsourcing them and
restoring the results inside the TEE. The non-linear layers are also kept
secure inside the TEE. ShadowNet's design ensures efficient transformation of
the weights and the subsequent restoration of the results. We build a ShadowNet
prototype based on TensorFlow Lite and evaluate it on five popular CNNs,
namely, MobileNet, ResNet-44, MiniVGG, ResNet-404, and YOLOv4-tiny. Our
evaluation shows that ShadowNet achieves strong security guarantees with
reasonable performance, offering a practical solution for secure on-device
model inference.
