  In distributed, or privacy-preserving learning, we are often given a set of
probabilistic models estimated from different local repositories, and asked to
combine them into a single model that gives efficient statistical estimation. A
simple method is to linearly average the parameters of the local models, which,
however, tends to be degenerate or not applicable on non-convex models, or
models with different parameter dimensions. One more practical strategy is to
generate bootstrap samples from the local models, and then learn a joint model
based on the combined bootstrap set. Unfortunately, the bootstrap procedure
introduces additional noise and can significantly deteriorate the performance.
In this work, we propose two variance reduction methods to correct the
bootstrap noise, including a weighted M-estimator that is both statistically
efficient and practically powerful. Both theoretical and empirical analysis is
provided to demonstrate our methods.
