  It is of fundamental importance to find algorithms obtaining optimal
performance for learning of statistical models in distributed and communication
limited systems. Aiming at characterizing the optimal strategies, we consider
learning of Gaussian Processes (GPs) in distributed systems as a pivotal
example. We first address a very basic problem: how many bits are required to
estimate the inner-products of Gaussian vectors across distributed machines?
Using information theoretic bounds, we obtain an optimal solution for the
problem which is based on vector quantization. Two suboptimal and more
practical schemes are also presented as substitute for the vector quantization
scheme. In particular, it is shown that the performance of one of the practical
schemes which is called per-symbol quantization is very close to the optimal
one. Schemes provided for the inner-product calculations are incorporated into
our proposed distributed learning methods for GPs. Experimental results show
that with spending few bits per symbol in our communication scheme, our
proposed methods outperform previous zero rate distributed GP learning schemes
such as Bayesian Committee Model (BCM) and Product of experts (PoE).
