  An existing hybrid MPI-OpenMP scheme is augmented with a CUDA-based fine
grain parallelization approach for multidimensional distributed Fourier
transforms, in a well-characterized pseudospectral fluid turbulence code.
Basics of the hybrid scheme are reviewed, and heuristics provided to show a
potential benefit of the CUDA implementation. The method draws heavily on the
CUDA runtime library to handle memory management, and on the cuFFT library for
computing local FFTs. The manner in which the interfaces are constructed to
these libraries, and ISO bindings utilized to facilitate platform portability,
are discussed. CUDA streams are implemented to overlap data transfer with cuFFT
computation. Testing with a baseline solver demonstrates significant aggregate
speed-up over the hybrid MPI-OpenMP solver by offloading to GPUs on an
NVLink-based test system. While the batch streamed approach provides little
benefit with NVLink, we see a performance gain of 30% when tuned for the
optimal number of streams on a PCIe-based system. It is found that strong GPU
scaling is ideal, or slightly better than ideal, in all cases. In addition to
speed-up measurements for the fiducial solver, we also consider several other
solvers with different numbers of transform operations and find that aggregate
speed-ups are nearly constant for all solvers.
