  Recent advances in randomized incremental methods for minimizing $L$-smooth
$\mu$-strongly convex finite sums have culminated in tight complexity of
$\tilde{O}((n+\sqrt{n L/\mu})\log(1/\epsilon))$ and $O(n+\sqrt{nL/\epsilon})$,
where $\mu>0$ and $\mu=0$, respectively, and $n$ denotes the number of
individual functions. Unlike incremental methods, stochastic methods for finite
sums do not rely on an explicit knowledge of which individual function is being
addressed at each iteration, and as such, must perform at least $\Omega(n^2)$
iterations to obtain $O(1/n^2)$-optimal solutions. In this work, we exploit the
finite noise structure of finite sums to derive a matching $O(n^2)$-upper bound
under the global oracle model, showing that this lower bound is indeed tight.
Following a similar approach, we propose a novel adaptation of SVRG which is
both \emph{compatible with stochastic oracles}, and achieves complexity bounds
of $\tilde{O}((n^2+n\sqrt{L/\mu})\log(1/\epsilon))$ and
$O(n\sqrt{L/\epsilon})$, for $\mu>0$ and $\mu=0$, respectively. Our bounds hold
w.h.p. and match in part existing lower bounds of
$\tilde{\Omega}(n^2+\sqrt{nL/\mu}\log(1/\epsilon))$ and
$\tilde{\Omega}(n^2+\sqrt{nL/\epsilon})$, for $\mu>0$ and $\mu=0$,
respectively.
