  Communication among devices in multi-GPU systems plays an important role in
terms of performance and scalability. In order to optimize an application,
programmers need to know the type and amount of the communication happening
among GPUs. Although there are prior works to gather this information in MPI
applications on distributed systems and multi-threaded applications on shared
memory systems, there is no tool that identifies communication among GPUs. Our
prior work, ComScribe, presents a point-to-point (P2P) communication detection
tool for GPUs sharing a common host. In this work, we extend ComScribe to
identify communication among GPUs for collective and P2P communication
primitives in NVIDIA's NCCL library. In addition to P2P communications,
collective communications are commonly used in HPC and AI workloads thus it is
important to monitor the induced data movement due to collectives. Our tool
extracts the size and the frequency of data transfers in an application and
visualizes them as a communication matrix. To demonstrate the tool in action,
we present communication matrices and some statistics for two applications
coming from machine translation and image classification domains.
