  Adapting model parameters to incoming streams of data is a crucial factor to
deep learning scalability. Interestingly, prior continual learning strategies
in online settings inadvertently anchor their updated parameters to a local
parameter subspace to remember old tasks, else drift away from the subspace and
forget. From this observation, we formulate a trade-off between constructing
multiple parameter modes and allocating tasks per mode. Mode-Optimized Task
Allocation (MOTA), our contributed adaptation strategy, trains multiple modes
in parallel, then optimizes task allocation per mode. We empirically
demonstrate improvements over baseline continual learning strategies and across
varying distribution shifts, namely sub-population, domain, and task shift.
