  The upcoming exascale era will provide a new generation of physics
simulations. These simulations will have a high spatiotemporal resolution,
which will impact the training of machine learning models since storing a high
amount of simulation data on disk is nearly impossible. Therefore, we need to
rethink the training of machine learning models for simulations for the
upcoming exascale era. This work presents an approach that trains a neural
network concurrently to a running simulation without storing data on a disk.
The training pipeline accesses the training data by in-memory streaming.
Furthermore, we apply methods from the domain of continual learning to enhance
the generalization of the model. We tested our pipeline on the training of a 3d
autoencoder trained concurrently to laser wakefield acceleration
particle-in-cell simulation. Furthermore, we experimented with various
continual learning methods and their effect on the generalization.
