  Incremental Learning (IL) is an interesting AI problem when the algorithm is
assumed to work on a budget. This is especially true when IL is modeled using a
deep learning approach, where two com- plex challenges arise due to limited
memory, which induces catastrophic forgetting and delays related to the
retraining needed in order to incorpo- rate new classes. Here we introduce
DeeSIL, an adaptation of a known transfer learning scheme that combines a fixed
deep representation used as feature extractor and learning independent shallow
classifiers to in- crease recognition capacity. This scheme tackles the two
aforementioned challenges since it works well with a limited memory budget and
each new concept can be added within a minute. Moreover, since no deep re-
training is needed when the model is incremented, DeeSIL can integrate larger
amounts of initial data that provide more transferable features. Performance is
evaluated on ImageNet LSVRC 2012 against three state of the art algorithms.
Results show that, at scale, DeeSIL performance is 23 and 33 points higher than
the best baseline when using the same and more initial data respectively.
