  Consider the problem of minimizing functions that are Lipschitz and strongly
convex, but not necessarily differentiable. We prove that after $T$ steps of
stochastic gradient descent, the error of the final iterate is $O(\log(T)/T)$
with high probability. We also construct a function from this class for which
the error of the final iterate of deterministic gradient descent is
$\Omega(\log(T)/T)$. This shows that the upper bound is tight and that, in this
setting, the last iterate of stochastic gradient descent has the same general
error rate (with high probability) as deterministic gradient descent. This
resolves both open questions posed by Shamir (2012).
  An intermediate step of our analysis proves that the suffix averaging method
achieves error $O(1/T)$ with high probability, which is optimal (for any
first-order optimization method). This improves results of Rakhlin (2012) and
Hazan and Kale (2014), both of which achieved error $O(1/T)$, but only in
expectation, and achieved a high probability error bound of $O(\log
\log(T)/T)$, which is suboptimal.
  We prove analogous results for functions that are Lipschitz and convex, but
not necessarily strongly convex or differentiable. After $T$ steps of
stochastic gradient descent, the error of the final iterate is
$O(\log(T)/\sqrt{T})$ with high probability, and there exists a function for
which the error of the final iterate of deterministic gradient descent is
$\Omega(\log(T)/\sqrt{T})$.
