  Graphics processing units (GPUs) are widely used in many high-performance
computing (HPC) applications such as imaging/video processing and training
deep-learning models in artificial intelligence. GPUs installed in HPC systems
are often heavily used, and GPU failures occur during HPC system operations.
Thus, the reliability of GPUs is of interest for the overall reliability of HPC
systems. The Cray XK7 Titan supercomputer was one of the top ten supercomputers
in the world. The failure event times of more than 30,000 GPUs in Titan were
recorded and previous data analysis suggested that the failure time of a GPU
may be affected by the GPU's connectivity location inside the supercomputer
among other factors. In this paper, we conduct in-depth statistical modeling of
GPU failure times to study the effect of location on GPU failures under
competing risks with covariates and spatially correlated random effects. In
particular, two major failure types of GPUs in Titan are considered. The
connectivity locations of cabinets are modeled as spatially correlated random
effects, and the positions of GPUs inside each cabinet are treated as
covariates. A Bayesian framework is used for statistical inference. We also
compare different methods of estimation such as the maximum likelihood, which
is implemented via an expectation-maximization algorithm. Our results provide
interesting insights into GPU failures in HPC systems.
