  Task-incremental learning involves the challenging problem of learning new
tasks continually, without forgetting past knowledge. Many approaches address
the problem by expanding the structure of a shared neural network as tasks
arrive, but struggle to grow optimally, without losing past knowledge. We
present a new framework, Learn to Bind and Grow, which learns a neural
architecture for a new task incrementally, either by binding with layers of a
similar task or by expanding layers which are more likely to conflict between
tasks. Central to our approach is a novel, interpretable, parameterization of
the shared, multi-task architecture space, which then enables computing
globally optimal architectures using Bayesian optimization. Experiments on
continual learning benchmarks show that our framework performs comparably with
earlier expansion based approaches and is able to flexibly compute multiple
optimal solutions with performance-size trade-offs.
