  Incremental gradient and incremental proximal methods are a fundamental class
of optimization algorithms used for solving finite sum problems, broadly
studied in the literature. Yet, without strong convexity, their convergence
guarantees have primarily been established for the ergodic (average) iterate.
Motivated by applications in continual learning, we obtain the first
convergence guarantees for the last iterate of both incremental gradient and
incremental proximal methods, in general convex smooth (for both) and convex
Lipschitz (for the proximal variants) settings. Our oracle complexity bounds
for the last iterate nearly match (i.e., match up to a square-root-log or a log
factor) the best known oracle complexity bounds for the average iterate, for
both classes of methods. We further obtain generalizations of our results to
weighted averaging of the iterates with increasing weights and for randomly
permuted ordering of updates. We study incremental proximal methods as a model
of continual learning with generalization and argue that large amount of
regularization is crucial to preventing catastrophic forgetting. Our results
generalize last iterate guarantees for incremental methods compared to state of
the art, as such results were previously known only for overparameterized
linear models, which correspond to convex quadratic problems with infinitely
many solutions.
