  Stochastic gradient descent is a canonical tool for addressing stochastic
optimization problems, and forms the bedrock of modern machine learning and
statistics. In this work, we seek to balance the fact that attenuating
step-size is required for exact asymptotic convergence with the fact that
constant step-size learns faster in finite time up to an error. To do so,
rather than fixing the mini-batch and the step-size at the outset, we propose a
strategy to allow parameters to evolve adaptively. Specifically, the batch-size
is set to be a piecewise-constant increasing sequence where the increase occurs
when a suitable error criterion is satisfied. Moreover, the step-size is
selected as that which yields the fastest convergence. The overall algorithm,
two scale adaptive (TSA) scheme, is developed for both convex and non-convex
stochastic optimization problems. It inherits the exact asymptotic convergence
of stochastic gradient method. More importantly, the optimal error decreasing
rate is achieved theoretically, as well as an overall reduction in
computational cost. Experimentally, we observe that TSA attains a favorable
tradeoff relative to standard SGD that fixes the mini-batch and the step-size,
or simply allowing one to increase or decrease respectively.
