  We consider a distributed learning setting where each agent/learner holds a
specific parametric model and data source. The goal is to integrate information
across a set of learners to enhance the prediction accuracy of a given learner.
A natural way to integrate information is to build a joint model across a group
of learners that shares common parameters of interest. However, the underlying
parameter sharing patterns across a set of learners may not be a priori known.
Misspecifying the parameter sharing patterns or the parametric model for each
learner often yields a biased estimation and degrades the prediction accuracy.
We propose a general method to integrate information across a set of learners
that is robust against misspecifications of both models and parameter sharing
patterns. The main crux is to sequentially incorporate additional learners that
can enhance the prediction accuracy of an existing joint model based on
user-specified parameter sharing patterns across a set of learners.
Theoretically, we show that the proposed method can data-adaptively select the
most suitable way of parameter sharing and thus enhance the predictive
performance of any particular learner of interest. Extensive numerical studies
show the promising performance of the proposed method.
