  This paper jointly considers privacy preservation and Byzantine-robustness in
decentralized learning. In a decentralized network, honest-but-curious agents
faithfully follow the prescribed algorithm, but expect to infer their
neighbors' private data from messages received during the learning process,
while dishonest-and-Byzantine agents disobey the prescribed algorithm, and
deliberately disseminate wrong messages to their neighbors so as to bias the
learning process. For this novel setting, we investigate a generic
privacy-preserving and Byzantine-robust decentralized stochastic gradient
descent (SGD) framework, in which Gaussian noise is injected to preserve
privacy and robust aggregation rules are adopted to counteract Byzantine
attacks. We analyze its learning error and privacy guarantee, discovering an
essential tradeoff between privacy preservation and Byzantine-robustness in
decentralized learning -- the learning error caused by defending against
Byzantine attacks is exacerbated by the Gaussian noise added to preserve
privacy. For a class of state-of-the-art robust aggregation rules, we give
unified analysis of the "mixing abilities". Building upon this analysis, we
reveal how the "mixing abilities" affect the tradeoff between privacy
preservation and Byzantine-robustness. The theoretical results provide
guidelines for achieving a favorable tradeoff with proper design of robust
aggregation rules. Numerical experiments are conducted and corroborate our
theoretical findings.
