  Realistic simulations in engineering or in the materials sciences can consume
enormous computing resources and thus require the use of massively parallel
supercomputers. The probability of a failure increases both with the runtime
and with the number of system components. For future exascale systems it is
therefore considered critical that strategies are developed to make software
resilient against failures. In this article, we present a scalable,
distributed, diskless, and resilient checkpointing scheme that can create and
recover snapshots of a partitioned simulation domain. We demonstrate the
efficiency and scalability of the checkpoint strategy for simulations with up
to $40$ billion computational cells executing on more than $400$ billion
floating point values. A checkpoint creation is shown to require only a few
seconds and the new checkpointing scheme scales almost perfectly up to more
than $260\,000$ ($2^{18}$) processes. To recover from a diskless checkpoint
during runtime, we realize the recovery algorithms using ULFM MPI. The
checkpointing mechanism is fully integrated in a state-of-the-art
high-performance multi-physics simulation framework. We demonstrate the
efficiency and robustness of the method with a realistic phase-field simulation
originating in the material sciences and with a lattice Boltzmann method
implementation.
