  Parallelization is a popular strategy for improving the performance of
iterative algorithms. Optimization methods are no exception: design of
efficient parallel optimization methods and tight analysis of their theoretical
properties are important research endeavors. While the minimax complexities are
well known for sequential optimization methods, the theory of parallel
optimization methods is less explored. In this paper, we propose a new protocol
that generalizes the classical oracle framework approach. Using this protocol,
we establish minimax complexities for parallel optimization methods that have
access to an unbiased stochastic gradient oracle with bounded variance. We
consider a fixed computation model characterized by each worker requiring a
fixed but worker-dependent time to calculate stochastic gradient. We prove
lower bounds and develop optimal algorithms that attain them. Our results have
surprising consequences for the literature of asynchronous optimization
methods.
