  Recent research identified a temporary performance drop on previously learned
tasks when transitioning to a new one. This drop is called the stability gap
and has great consequences for continual learning: it complicates the direct
employment of continually learning since the worse-case performance at
task-boundaries is dramatic, it limits its potential as an energy-efficient
training paradigm, and finally, the stability drop could result in a reduced
final performance of the algorithm. In this paper, we show that the stability
gap also occurs when applying joint incremental training of homogeneous tasks.
In this scenario, the learner continues training on the same data distribution
and has access to all data from previous tasks. In addition, we show that in
this scenario, there exists a low-loss linear path to the next minima, but that
SGD optimization does not choose this path. We perform further analysis
including a finer batch-wise analysis which could provide insights towards
potential solution directions.
