  For minimizing a strongly convex objective function subject to linear
inequality constraints, we consider a penalty approach that allows one to
utilize stochastic methods for problems with a large number of constraints
and/or objective function terms. We provide upper bounds on the distance
between the solutions to the original constrained problem and the penalty
reformulations, guaranteeing the convergence of the proposed approach. We give
a nested accelerated stochastic gradient method and propose a novel way for
updating the smoothness parameter of the penalty function and the step-size.
The proposed algorithm requires at most $\tilde O(1/\sqrt{\epsilon})$ expected
stochastic gradient iterations to produce a solution within an expected
distance of $\epsilon$ to the optimal solution of the original problem, which
is the best complexity for this problem class to the best of our knowledge. We
also show how to query an approximate dual solution after stochastically
solving the penalty reformulations, leading to results on the convergence of
the duality gap. Moreover, the nested structure of the algorithm and upper
bounds on the distance to the optimal solutions allows one to safely eliminate
constraints that are inactive at an optimal solution throughout the algorithm,
which leads to improved complexity results. Finally, we present computational
results that demonstrate the effectiveness and robustness of our algorithm.
