  Developing efficient GPU kernels can be difficult because of the complexity
of GPU architectures and programming models. Existing performance tools only
provide coarse-grained suggestions at the kernel level, if any. In this paper,
we describe GPA, a performance advisor for NVIDIA GPUs that suggests potential
code optimization opportunities at a hierarchy of levels, including individual
lines, loops, and functions. To relieve users of the burden of interpreting
performance counters and analyzing bottlenecks, GPA uses data flow analysis to
approximately attribute measured instruction stalls to their root causes and
uses information about a program's structure and the GPU to match inefficiency
patterns with suggestions for optimization. To quantify each suggestion's
potential benefits, we developed PC sampling-based performance models to
estimate its speedup. Our experiments with benchmarks and applications show
that GPA provides an insightful report to guide performance optimization. Using
GPA, we obtained speedups on a Volta V100 GPU ranging from 1.01$\times$ to
3.53$\times$, with a geometric mean of 1.22$\times$.
