  We propose an efficient distributed out-of-memory implementation of the
Non-negative Matrix Factorization (NMF) algorithm for heterogeneous
high-performance-computing (HPC) systems. The proposed implementation is based
on prior work on NMFk, which can perform automatic model selection and extract
latent variables and patterns from data. In this work, we extend NMFk by adding
support for dense and sparse matrix operation on multi-node, multi-GPU systems.
The resulting algorithm is optimized for out-of-memory (OOM) problems where the
memory required to factorize a given matrix is greater than the available GPU
memory. Memory complexity is reduced by batching/tiling strategies, and sparse
and dense matrix operations are significantly accelerated with GPU cores (or
tensor cores when available). Input/Output (I/O) latency associated with batch
copies between host and device is hidden using CUDA streams to overlap data
transfers and compute asynchronously, and latency associated with collective
communications (both intra-node and inter-node) is reduced using optimized
NVIDIA Collective Communication Library NCCL based communicators. Benchmark
results show significant improvement, from 32X to 76x speedup, with the new
implementation using GPUs over the CPU-based NMFk. Good weak scaling was
demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000
GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size
sparse matrix of density 10e-6.
