  Manufacturers have been developing new graphics processing unit (GPU) nodes
with large capacity, high bandwidth memory and very high bandwidth intra-node
interconnects. This enables moving large amounts of data between GPUs on the
same node at low cost. However, small packet bandwidths and latencies have not
decreased which makes global dot products expensive. These characteristics
favor a new kind of problem decomposition called "equation decomposition"
rather than traditional domain decomposition. In this approach, each GPU is
assigned one equation set to solve in parallel so that the frequent and
expensive dot product synchronization points in traditional distributed linear
solvers are eliminated. In exchange, the method involves infrequent movement of
state variables over the high bandwidth, intra-node interconnects. To test this
theory, our flagship code Multiphase Flow with Interphase eXchanges (MFiX) was
ported to TensorFlow. This new product is known as MFiX-AI and can produce near
identical results to the original version of MFiX with significant acceleration
in multiphase particle-in-cell (MP-PIC) simulations. The performance of a
single node with 4 NVIDIA A100s connected over NVLINK 2.0 was shown to be
competitive to 1000 CPU cores (25 nodes) on the JOULE 2.0 supercomputer,
leading to an energy savings of up to 90%. This is a substantial performance
benefit for small- to intermediate-sized problems. This benefit is expected to
grow as GPU nodes become more powerful. Further, MFiX-AI is poised to accept
native artificial intelligence/machine learning models for further acceleration
and development.
