  This paper explores strategies to transform an existing CPU-based
high-performance computational fluid dynamics solver, HyPar, for compressible
flow simulations on emerging exascale heterogeneous (CPU+GPU) computing
platforms. The scientific motivation for developing a GPU-enhanced version of
HyPar is to simulate canonical turbulent flows at the highest resolution
possible on such platforms. We show that optimizing memory operations and
thread blocks results in 200x speedup of computationally intensive kernels
compared with a CPU core. Using multiple GPUs and CUDA-aware MPI communication,
we demonstrate both strong and weak scaling of our GPU-based HyPar
implementation on the NVIDIA Volta V100 GPUs. We simulate the decay of
homogeneous isotropic turbulence in a triply periodic box on grids with up to
$1024^3$ points (5.3 billion degrees of freedom) and on up to 1,024 GPUs. We
compare the wall times for CPU-only and CPU+GPU simulations. The results
presented in the paper are obtained on the Summit and Lassen supercomputers at
Oak Ridge and Lawrence Livermore National Laboratories, respectively.
