  We present a set of fault injection experiments performed on the ACES
(LANL/SNL) Cray XE supercomputer Cielo. We use this experimental campaign to
improve the understanding of failure causes and propagation that we observed in
the field failure data analysis of NCSA's Blue Waters. We use the data
collected from the logs and from network performance counter data 1) to
characterize the fault-error-failure sequence and recovery mechanisms in the
Gemini network and in the Cray compute nodes, 2) to understand the impact of
failures on the system and the user applications at different scale, and 3) to
identify and recreate fault scenarios that induce unrecoverable failures, in
order to create new tests for system and application design. The faults were
injected through special input commands to bring down network links,
directional connections, nodes, and blades. We present extensions that will be
needed to apply our methodologies of injection and analysis to the Cray XC
(Aries) systems.
