  Informative gradients are often lost in large batch updates. We propose a
robust mechanism to reinforce the sparse components within a random batch of
data points. A finite queue of online gradients is used to determine their
expected instantaneous statistics. We propose a function to measure the
scarcity of incoming gradients using these statistics and establish the
theoretical ground of this mechanism. To minimize conflicting components within
large mini-batches, samples are grouped with aligned objectives by clustering
based on inherent feature space. Sparsity is measured for each centroid and
weighted accordingly. A strong intuitive criterion to squeeze out redundant
information from each cluster is the backbone of the system. It makes rare
information indifferent to aggressive momentum also exhibits superior
performance with larger mini-batch horizon. The effective length of the queue
kept variable to follow the local loss pattern. The contribution of our method
is to restore intra-mini-batch diversity at the same time widening the optimal
batch boundary. Both of these collectively drive it deeper towards the minima.
Our method has shown superior performance for CIFAR10, MNIST, and Reuters News
category dataset compared to mini-batch gradient descent.
