  Matrix decomposition is one of the fundamental tools to discover knowledge
from big data generated by modern applications. However, it is still
inefficient or infeasible to process very big data using such a method in a
single machine. Moreover, big data are often distributedly collected and stored
on different machines. Thus, such data generally bear strong heterogeneous
noise. It is essential and useful to develop distributed matrix decomposition
for big data analytics. Such a method should scale up well, model the
heterogeneous noise, and address the communication issue in a distributed
system. To this end, we propose a distributed Bayesian matrix decomposition
model (DBMD) for big data mining and clustering. Specifically, we adopt three
strategies to implement the distributed computing including 1) the accelerated
gradient descent, 2) the alternating direction method of multipliers (ADMM),
and 3) the statistical inference. We investigate the theoretical convergence
behaviors of these algorithms. To address the heterogeneity of the noise, we
propose an optimal plug-in weighted average that reduces the variance of the
estimation. Synthetic experiments validate our theoretical results, and
real-world experiments show that our algorithms scale up well to big data and
achieves superior or competing performance compared to other distributed
methods.
