  Learning from non-stationary data streams, also called Task-Free Continual
Learning (TFCL) remains challenging due to the absence of explicit task
information. Although recently some methods have been proposed for TFCL, they
lack theoretical guarantees. Moreover, forgetting analysis during TFCL was not
studied theoretically before. This paper develops a new theoretical analysis
framework which provides generalization bounds based on the discrepancy
distance between the visited samples and the entire information made available
for training the model. This analysis gives new insights into the forgetting
behaviour in classification tasks. Inspired by this theoretical model, we
propose a new approach enabled by the dynamic component expansion mechanism for
a mixture model, namely the Online Discrepancy Distance Learning (ODDL). ODDL
estimates the discrepancy between the probabilistic representation of the
current memory buffer and the already accumulated knowledge and uses it as the
expansion signal to ensure a compact network architecture with optimal
performance. We then propose a new sample selection approach that selectively
stores the most relevant samples into the memory buffer through the
discrepancy-based measure, further improving the performance. We perform
several TFCL experiments with the proposed methodology, which demonstrate that
the proposed approach achieves the state of the art performance.
