  Recent years have seen a surge in deep learning approaches to accelerate
numerical solvers, which provide faithful but computationally intensive
simulations of the physical world. These deep surrogates are generally trained
in a supervised manner from limited amounts of data slowly generated by the
same solver they intend to accelerate. We propose an open-source framework that
enables the online training of these models from a large ensemble run of
simulations. It leverages multiple levels of parallelism to generate rich
datasets. The framework avoids I/O bottlenecks and storage issues by directly
streaming the generated data. A training reservoir mitigates the inherent bias
of streaming while maximizing GPU throughput. Experiment on training a fully
connected network as a surrogate for the heat equation shows the proposed
approach enables training on 8TB of data in 2 hours with an accuracy improved
by 47% and a batch throughput multiplied by 13 compared to a traditional
offline procedure.
