  Reliability is a serious concern for future extreme-scale high-performance
computing (HPC) systems. While the HPC community has developed various
resilience solutions, the solution space remains fragmented. There are no
formal methods and metrics to integrate the various HPC resilience techniques
into composite solutions, nor are there methods to holistically evaluate the
adequacy and efficacy of such solutions in terms of their protection coverage,
and their performance & power efficiency characteristics. In this paper, we
develop a structured approach to the design, evaluation and optimization of HPC
resilience using the concept of design patterns. A design pattern is a general
repeatable solution to a commonly occurring problem. We identify the problems
caused by various types of faults, errors and failures in HPC systems and the
techniques used to deal with these events. Each well-known solution that
addresses a specific HPC resilience challenge is described in the form of a
pattern. We develop a complete catalog of such resilience design patterns,
which may be used as essential building blocks when designing and deploying
resilience solutions. We also develop a design framework that enhances a
designer's understanding the opportunities for integrating multiple patterns
across layers of the system stack and the important constraints during
implementation of the individual patterns. It is also useful for defining
mechanisms and interfaces to coordinate flexible fault management across
hardware and software components. The overall goal of this work is to establish
a systematic methodology for the design and evaluation of resilience
technologies in extreme-scale HPC systems that keep scientific applications
running to a correct solution in a timely and cost-efficient manner despite
frequent faults, errors, and failures of various types.
