  Deep ensemble learning has been shown to improve accuracy by training
multiple neural networks and averaging their outputs. Ensemble learning has
also been suggested to defend against membership inference attacks that
undermine privacy. In this paper, we empirically demonstrate a trade-off
between these two goals, namely accuracy and privacy (in terms of membership
inference attacks), in deep ensembles. Using a wide range of datasets and model
architectures, we show that the effectiveness of membership inference attacks
increases when ensembling improves accuracy. We analyze the impact of various
factors in deep ensembles and demonstrate the root cause of the trade-off.
Then, we evaluate common defenses against membership inference attacks based on
regularization and differential privacy. We show that while these defenses can
mitigate the effectiveness of membership inference attacks, they simultaneously
degrade ensemble accuracy. We illustrate similar trade-off in more advanced and
state-of-the-art ensembling techniques, such as snapshot ensembles and
diversified ensemble networks. Finally, we propose a simple yet effective
defense for deep ensembles to break the trade-off and, consequently, improve
the accuracy and privacy, simultaneously.
