  In class incremental learning, neural networks typically suffer from
catastrophic forgetting. We show that an MLP featuring a sparse activation
function and an adaptive learning rate optimizer can compete with established
regularization techniques in the Split-MNIST task. We highlight the
effectiveness of the Adaptive SwisH (ASH) activation function in this context
and introduce a novel variant, Hard Adaptive SwisH (Hard ASH) to further
enhance the learning retention.
