  Due to the recent announcement of the Frontier supercomputer, many scientific
application developers are working to make their applications compatible with
AMD architectures (CPU-GPU), which means moving away from the traditional CPU
and NVIDIA-GPU systems. Due to the current limitations of profiling tools for
AMD GPUs, this shift leaves a void in how to measure application performance on
AMD GPUs. In this paper, we design an instruction roofline model for AMD GPUs
using AMD's ROCProfiler and a benchmarking tool, BabelStream (the HIP
implementation), as a way to measure an application's performance in
instructions and memory transactions on new AMD hardware. Specifically, we
create instruction roofline models for a case study scientific application,
PIConGPU, an open source particle-in-cell (PIC) simulations application used
for plasma and laser-plasma physics on the NVIDIA V100, AMD Radeon Instinct
MI60, and AMD Instinct MI100 GPUs. When looking at the performance of multiple
kernels of interest in PIConGPU we find that although the AMD MI100 GPU
achieves a similar, or better, execution time compared to the NVIDIA V100 GPU,
profiling tool differences make comparing performance of these two
architectures hard. When looking at execution time, GIPS, and instruction
intensity, the AMD MI60 achieves the worst performance out of the three GPUs
used in this work.
