  Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop
and Adam) have been widely used in training deep neural networks. Despite their
fast convergence, they can generalize worse than stochastic gradient descent.
In this paper, by revisiting the design of Adagrad, we propose to split the
network parameters into blocks, and use a blockwise adaptive stepsize.
Intuitively, blockwise adaptivity is less aggressive than adaptivity to
individual coordinates, and can have a better balance between adaptivity and
generalization. We show theoretically that the proposed blockwise adaptive
gradient descent has comparable convergence rate as its counterpart with
coordinate-wise adaptive stepsize, but is faster up to some constant. We also
study its uniform stability and show that blockwise adaptivity can lead to
lower generalization error than coordinate-wise adaptivity. Experimental
results show that blockwise adaptive gradient descent converges faster and
improves generalization performance over Nesterov's accelerated gradient and
Adam.
