  Personalized learning is a proposed approach to address the problem of data
heterogeneity in collaborative machine learning. In a decentralized setting,
the two main challenges of personalization are client clustering and data
privacy. In this paper, we address these challenges by developing P4
(Personalized Private Peer-to-Peer) a method that ensures that each client
receives a personalized model while maintaining differential privacy guarantee
of each client's local dataset during and after the training. Our approach
includes the design of a lightweight algorithm to identify similar clients and
group them in a private, peer-to-peer (P2P) manner. Once grouped, we develop
differentially-private knowledge distillation for clients to co-train with
minimal impact on accuracy. We evaluate our proposed method on three benchmark
datasets (FEMNIST or Federated EMNIST, CIFAR-10 and CIFAR-100) and two
different neural network architectures (Linear and CNN-based networks) across a
range of privacy parameters. The results demonstrate the potential of P4, as it
outperforms the state-of-the-art of differential private P2P by up to 40
percent in terms of accuracy. We also show the practicality of P4 by
implementing it on resource constrained devices, and validating that it has
minimal overhead, e.g., about 7 seconds to run collaborative training between
two clients.
