  Performing machine learning tasks in mobile applications yields a challenging
conflict of interest: highly sensitive client information (e.g., speech data)
should remain private while also the intellectual property of service providers
(e.g., model parameters) must be protected. Cryptographic techniques offer
secure solutions for this, but have an unacceptable overhead and moreover
require frequent network interaction. In this work, we design a practically
efficient hardware-based solution. Specifically, we build Offline Model Guard
(OMG) to enable privacy-preserving machine learning on the predominant mobile
computing platform ARM - even in offline scenarios. By leveraging a trusted
execution environment for strict hardware-enforced isolation from other system
components, OMG guarantees privacy of client data, secrecy of provided models,
and integrity of processing algorithms. Our prototype implementation on an ARM
HiKey 960 development board performs privacy-preserving keyword recognition
using TensorFlow Lite for Microcontrollers in real time.
