  This paper investigates capabilities of Privacy-Preserving Deep Learning
(PPDL) mechanisms against various forms of privacy attacks. First, we propose
to quantitatively measure the trade-off between model accuracy and privacy
losses incurred by reconstruction, tracing and membership attacks. Second, we
formulate reconstruction attacks as solving a noisy system of linear equations,
and prove that attacks are guaranteed to be defeated if condition (2) is
unfulfilled. Third, based on theoretical analysis, a novel Secret Polarization
Network (SPN) is proposed to thwart privacy attacks, which pose serious
challenges to existing PPDL methods. Extensive experiments showed that model
accuracies are improved on average by 5-20% compared with baseline mechanisms,
in regimes where data privacy are satisfactorily protected.
