  Cloud deep learning platforms provide cost-effective deep neural network
(DNN) training for customers who lack computation resources. However, cloud
systems are often untrustworthy and vulnerable to attackers, leading to growing
concerns about model privacy. Recently, researchers have sought to protect data
privacy in deep learning by leveraging CPU trusted execution environments
(TEEs), which minimize the use of cryptography, but existing works failed to
simultaneously utilize the computational resources of GPUs to assist in
training and prevent model leakage. This paper presents Tempo, the first
cloud-based deep learning system that cooperates with TEE and distributed GPUs
for efficient DNN training with model confidentiality preserved. To tackle the
challenge of preserving privacy while offloading linear algebraic operations
from TEE to GPUs for efficient batch computation, we introduce a customized
permutation-based obfuscation algorithm to blind both inputs and model
parameters. An optimization mechanism that reduces encryption operations is
proposed for faster weight updates during backpropagation to speed up training.
We implement Tempo and evaluate it with both training and inference for two
prevalent DNNs. Empirical results indicate that Tempo outperforms baselines and
offers sufficient privacy protection.
