  Two types of low cost-per-iteration gradient descent methods have been
extensively studied in parallel. One is online or stochastic gradient descent
(OGD/SGD), and the other is randomzied coordinate descent (RBCD). In this
paper, we combine the two types of methods together and propose online
randomized block coordinate descent (ORBCD). At each iteration, ORBCD only
computes the partial gradient of one block coordinate of one mini-batch
samples. ORBCD is well suited for the composite minimization problem where one
function is the average of the losses of a large number of samples and the
other is a simple regularizer defined on high dimensional variables. We show
that the iteration complexity of ORBCD has the same order as OGD or SGD. For
strongly convex functions, by reducing the variance of stochastic gradients, we
show that ORBCD can converge at a geometric rate in expectation, matching the
convergence rate of SGD with variance reduction and RBCD.
