  Data privacy is of great concern in cloud machine-learning service platforms,
when sensitive data are exposed to service providers. While private computing
environments (e.g., secure enclaves), and cryptographic approaches (e.g.,
homomorphic encryption) provide strong privacy protection, their computing
performance still falls short compared to cloud GPUs. To achieve privacy
protection with high computing performance, we propose Delta, a new private
training and inference framework, with comparable model performance as
non-private centralized training. Delta features two asymmetric data flows: the
main information-sensitive flow and the residual flow. The main part flows into
a small model while the residuals are offloaded to a large model. Specifically,
Delta embeds the information-sensitive representations into a low-dimensional
space while pushing the information-insensitive part into high-dimension
residuals. To ensure privacy protection, the low-dimensional
information-sensitive part is secured and fed to a small model in a private
environment. On the other hand, the residual part is sent to fast cloud GPUs,
and processed by a large model. To further enhance privacy and reduce the
communication cost, Delta applies a random binary quantization technique along
with a DP-based technique to the residuals before sharing them with the public
platform. We theoretically show that Delta guarantees differential privacy in
the public environment and greatly reduces the complexity in the private
environment. We conduct empirical analyses on CIFAR-10, CIFAR-100 and ImageNet
datasets and ResNet-18 and ResNet-34, showing that Delta achieves strong
privacy protection, fast training, and inference without significantly
compromising the model utility.
