  Deep Neural Network (DNN), one of the most powerful machine learning
algorithms, is increasingly leveraged to overcome the bottleneck of effectively
exploring and analyzing massive data to boost advanced scientific development.
It is not a surprise that cloud computing providers offer the cloud-based DNN
as an out-of-the-box service. Though there are some benefits from the
cloud-based DNN, the interaction mechanism among two or multiple entities in
the cloud inevitably induces new privacy risks. This survey presents the most
recent findings of privacy attacks and defenses appeared in cloud-based neural
network services. We systematically and thoroughly review privacy attacks and
defenses in the pipeline of cloud-based DNN service, i.e., data manipulation,
training, and prediction. In particular, a new theory, called cloud-based ML
privacy game, is extracted from the recently published literature to provide a
deep understanding of state-of-the-art research. Finally, the challenges and
future work are presented to help researchers to continue to push forward the
competitions between privacy attackers and defenders.
