  In this paper, we studied a buffered mini-batch gradient descent (BMGD)
algorithm for training complex model on massive datasets. The algorithm studied
here is designed for fast training on a GPU-CPU system, which contains two
steps: the buffering step and the computation step. In the buffering step, a
large batch of data (i.e., a buffer) are loaded from the hard drive to the
graphical memory of GPU. In the computation step, a standard mini-batch
gradient descent(MGD) algorithm is applied to the buffered data. Compared to
traditional MGD algorithm, the proposed BMGD algorithm can be more efficient
for two reasons.First, the BMGD algorithm uses the buffered data for multiple
rounds of gradient update, which reduces the expensive communication cost from
the hard drive to GPU memory. Second, the buffering step can be executed in
parallel so that the GPU does not have to stay idle when loading new data. We
first investigate the theoretical properties of BMGD algorithms under a linear
regression setting.The analysis is then extended to the Polyak-Lojasiewicz loss
function class. The theoretical claims about the BMGD algorithm are numerically
verified by simulation studies. The practical usefulness of the proposed method
is demonstrated by three image-related real data analysis.
