  Variational Autoencoders (VAEs) suffer from degenerated performance, when
learning several successive tasks. This is caused by catastrophic forgetting.
In order to address the knowledge loss, VAEs are using either Generative Replay
(GR) mechanisms or Expanding Network Architectures (ENA). In this paper we
study the forgetting behaviour of VAEs using a joint GR and ENA methodology, by
deriving an upper bound on the negative marginal log-likelihood. This
theoretical analysis provides new insights into how VAEs forget the previously
learnt knowledge during lifelong learning. The analysis indicates the best
performance achieved when considering model mixtures, under the ENA framework,
where there are no restrictions on the number of components. However, an
ENA-based approach may require an excessive number of parameters. This
motivates us to propose a novel Dynamic Expansion Graph Model (DEGM). DEGM
expands its architecture, according to the novelty associated with each new
databases, when compared to the information already learnt by the network from
previous tasks. DEGM training optimizes knowledge structuring, characterizing
the joint probabilistic representations corresponding to the past and more
recently learned tasks. We demonstrate that DEGM guarantees optimal performance
for each task while also minimizing the required number of parameters.
Supplementary materials (SM) and source code are available in
https://github.com/dtuzi123/Expansion-Graph-Model.
