  This paper is focused on improving multi-GPU performance of a research CFD
code on structured grids. MPI and OpenACC directives are used to scale the code
up to 16 GPUs. This paper shows that using 16 P100 GPUs and 16 V100 GPUs can be
30$\times$ and 70$\times$ faster than 16 Xeon CPU E5-2680v4 cores for three
different test cases, respectively. A series of performance issues related to
the scaling for the multi-block CFD code are addressed by applying various
optimizations. Performance optimizations such as the pack/unpack message
method, removing temporary arrays as arguments to procedure calls, allocating
global memory for limiters and connected boundary data, reordering non-blocking
MPI I\_send/I\_recv and Wait calls, reducing unnecessary implicit derived type
member data movement between the host and the device and the use of GPUDirect
can improve the compute utilization, memory throughput, and asynchronous
progression in the multi-block CFD code using modern programming features.
