  Distributed learning of probabilistic models from multiple data repositories
with minimum communication is increasingly important. We study a simple
communication-efficient learning framework that first calculates the local
maximum likelihood estimates (MLE) based on the data subsets, and then combines
the local MLEs to achieve the best possible approximation to the global MLE
given the whole dataset. We study this framework's statistical properties,
showing that the efficiency loss compared to the global setting relates to how
much the underlying distribution families deviate from full exponential
families, drawing connection to the theory of information loss by Fisher, Rao
and Efron. We show that the "full-exponential-family-ness" represents the lower
bound of the error rate of arbitrary combinations of local MLEs, and is
achieved by a KL-divergence-based combination method but not by a more common
linear combination method. We also study the empirical properties of both
methods, showing that the KL method significantly outperforms linear
combination in practical settings with issues such as model misspecification,
non-convexity, and heterogeneous data partitions.
