  Secure multi-party computation-based machine learning, referred to as MPL,
has become an important technology to utilize data from multiple parties with
privacy preservation. While MPL provides rigorous security guarantees for the
computation process, the models trained by MPL are still vulnerable to attacks
that solely depend on access to the models. Differential privacy could help to
defend against such attacks. However, the accuracy loss brought by differential
privacy and the huge communication overhead of secure multi-party computation
protocols make it highly challenging to balance the 3-way trade-off between
privacy, efficiency, and accuracy.
  In this paper, we are motivated to resolve the above issue by proposing a
solution, referred to as PEA (Private, Efficient, Accurate), which consists of
a secure DPSGD protocol and two optimization methods. First, we propose a
secure DPSGD protocol to enforce DPSGD in secret sharing-based MPL frameworks.
Second, to reduce the accuracy loss led by differential privacy noise and the
huge communication overhead of MPL, we propose two optimization methods for the
training process of MPL: (1) the data-independent feature extraction method,
which aims to simplify the trained model structure; (2) the local data-based
global model initialization method, which aims to speed up the convergence of
the model training. We implement PEA in two open-source MPL frameworks:
TF-Encrypted and Queqiao. The experimental results on various datasets
demonstrate the efficiency and effectiveness of PEA. E.g. when ${\epsilon}$ =
2, we can train a differentially private classification model with an accuracy
of 88% for CIFAR-10 within 7 minutes under the LAN setting. This result
significantly outperforms the one from CryptGPU, one SOTA MPL framework: it
costs more than 16 hours to train a non-private deep neural network model on
CIFAR-10 with the same accuracy.
