  Performance optimization can be a daunting task especially as the hardware
architecture becomes more and more complex. This paper takes a kernel from the
Materials Science code BerkeleyGW, and demonstrates a few performance analysis
and optimization techniques. Despite challenges such as high register usage,
low occupancy, complex data access patterns, and the existence of several
long-latency instructions, we have achieved 3.7 TFLOP/s of double-precision
performance on an NVIDIA V100 GPU, with 8 optimization steps. This is 55% of
the theoretical peak, 6.7 TFLOP/s, at nominal frequency 1312 MHz, and 70% of
the more customized peak based on our 58% FMA ratio, 5.3 TFLOP/s. An array of
techniques used to analyze this OpenACC kernel and optimize its performance are
shown, including the use of hierarchical Roofline performance model and the
performance tool Nsight Compute. This kernel exhibits computational
characteristics that are commonly seen in many high-performance computing (HPC)
applications, and are expected to be very helpful to a general audience of HPC
developers and computational scientists, as they pursue more performance on
NVIDIA GPUs.
