  Scientific applications have long embraced the MPI as the environment of
choice to execute on large distributed systems. The User-Level Failure
Mitigation (ULFM) specification extends the MPI standard to address resilience
and enable MPI applications to restore their communication capability after a
failure. This works builds upon the wide body of experience gained in the field
to eliminate a gap between current practice and the ideal, more asynchronous,
recovery model in which the fault tolerance activities of multiple components
can be carried out simultaneously and overlap. This work proposes to: (1)
provide the required consistency in fault reporting to applications (i.e.,
enable an application to assess the success of a computational phase without
incurring an unacceptable performance hit); (2) bring forward the building
blocks that permit the effective scoping of fault recovery in an application,
so that independent components in an application can recover without
interfering with each other, and separate groups of processes in the
application can recover independently or in unison; and (3) overlap recovery
activities necessary to restore the consistency of the system (e.g., eviction
of faulty processes from the communication group) with application recovery
activities (e.g., dataset restoration from checkpoints).
