  To gain a better performance, many researchers put more computing resource
into an application. However, in the AI area, there is still a lack of a
successful large-scale machine learning training application: The scalability
and performance reproducibility of parallel machine learning training algorithm
are limited and there are a few pieces of research focusing on why these
indexes are limited but there are very few research efforts explaining the
reasons in essence. In this paper, we propose that the sample difference in
dataset plays a more prominent role in parallel machine learning algorithm
scalability. Dataset characters can measure sample difference. These characters
include the variance of the sample in a dataset, sparsity, sample diversity and
similarity in sampling sequence. To match our proposal, we choose four kinds of
parallel machine learning training algorithms as our research objects: (1)
Asynchronous parallel SGD algorithm (Hogwild! algorithm) (2) Parallel model
average SGD algorithm (Mini-batch SGD algorithm) (3) Decenterilization
optimization algorithm, (4) Dual Coordinate Optimization (DADM algorithm).
These algorithms cover different types of machine learning optimization
algorithms. We present the analysis of their convergence proof and design
experiments. Our results show that the characters datasets decide the
scalability of the machine learning algorithm. What is more, there is an upper
bound of parallel scalability for machine learning algorithms.
