  To meet the standard of differential privacy, noise is usually added into the
original data, which inevitably deteriorates the predicting performance of
subsequent learning algorithms. In this paper, motivated by the success of
improving predicting performance by ensemble learning, we propose to enhance
privacy-preserving logistic regression by stacking. We show that this can be
done either by sample-based or feature-based partitioning. However, we prove
that when privacy-budgets are the same, feature-based partitioning requires
fewer samples than sample-based one, and thus likely has better empirical
performance. As transfer learning is difficult to be integrated with a
differential privacy guarantee, we further combine the proposed method with
hypothesis transfer learning to address the problem of learning across
different organizations. Finally, we not only demonstrate the effectiveness of
our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply
it into a real application of cross-organizational diabetes prediction from
RUIJIN data set, where privacy is of significant concern.
