  Uncertainty quantification for forward and inverse problems is a central
challenge across physical and biomedical disciplines. We address this challenge
for the problem of modeling subsurface flow at the Hanford Site by combining
stochastic computational models with observational data using physics-informed
GAN models. The geographic extent, spatial heterogeneity, and multiple
correlation length scales of the Hanford Site require training a
computationally intensive GAN model to thousands of dimensions. We develop a
hierarchical scheme for exploiting domain parallelism, map discriminators and
generators to multiple GPUs, and employ efficient communication schemes to
ensure training stability and convergence. We developed a highly optimized
implementation of this scheme that scales to 27,500 NVIDIA Volta GPUs and 4584
nodes on the Summit supercomputer with a 93.1% scaling efficiency, achieving
peak and sustained half-precision rates of 1228 PF/s and 1207 PF/s.
