  Distributed machine learning (ML) is a modern computation paradigm that
divides its workload into independent tasks that can be simultaneously achieved
by multiple machines (i.e., agents) for better scalability. However, a typical
distributed system is usually implemented with a central server that collects
data statistics from multiple independent machines operating on different
subsets of data to build a global analytic model. This centralized
communication architecture however exposes a single choke point for operational
failure and places severe bottlenecks on the server's communication and
computation capacities as it has to process a growing volume of communication
from a crowd of learning agents. To mitigate these bottlenecks, this paper
introduces a novel Collective Online Learning Gaussian Process framework for
massive distributed systems that allows each agent to build its local model,
which can be exchanged and combined efficiently with others via peer-to-peer
communication to converge on a global model of higher quality. Finally, our
empirical results consistently demonstrate the efficiency of our framework on
both synthetic and real-world datasets.
