  In this paper, we consider a class of finite-sum convex optimization problems
whose objective function is given by the summation of $m$ ($\ge 1$) smooth
components together with some other relatively simple terms. We first introduce
a deterministic primal-dual gradient (PDG) method that can achieve the optimal
black-box iteration complexity for solving these composite optimization
problems using a primal-dual termination criterion. Our major contribution is
to develop a randomized primal-dual gradient (RPDG) method, which needs to
compute the gradient of only one randomly selected smooth component at each
iteration, but can possibly achieve better complexity than PDG in terms of the
total number of gradient evaluations. More specifically, we show that the total
number of gradient evaluations performed by RPDG can be ${\cal O} (\sqrt{m})$
times smaller, both in expectation and with high probability, than those
performed by deterministic optimal first-order methods under favorable
situations. We also show that the complexity of the RPDG method is not
improvable by developing a new lower complexity bound for a general class of
randomized methods for solving large-scale finite-sum convex optimization
problems. Moreover, through the development of PDG and RPDG, we introduce a
novel game-theoretic interpretation for these optimal methods for convex
optimization.
