  Understanding and visualizing the full-stack performance trade-offs and
interplay between HPC applications, MPI libraries, the communication fabric,
and the file system is a challenging endeavor. Designing a holistic profiling
and visualization method for HPC communication networks is challenging since
different levels of communication coexist and interact with each other on the
communication fabric. A breakdown of traffic is essential to understand the
interplay of different layers along with the application's communication
behavior without losing a general view of network traffic. Unfortunately,
existing profiling tools are disjoint and either focus on only profiling and
visualizing a few levels of the HPC stack, which limits the insights they can
provide, or they provide extremely detailed information which necessitates a
steep learning curve to understand. We target our profiling tool visualization
to provide holistic and real-time insights into HPC communication stacks.
  In this paper, we propose and implement our visualization methods to enable
holistic insight for representing the cross-stack metrics. Moreover, we propose
and implement a low-overhead I/O profiling inside the communication library,
collect and store the profiling information, and then study the correlation and
evaluation of I/O traffic with MPI communication using a cross-stack approach
by INAM. Through experimental evaluations and use cases, we demonstrate novel
benefits of our cross-stack communication analysis in real-time to detect
bottlenecks and understand communication performance.
