  It is commonly assumed that the end-to-end networking performance of edge
offloading is purely dictated by that of the network connectivity between end
devices and edge computing facilities, where ongoing innovation in 5G/6G
networking can help. However, with the growing complexity of edge-offloaded
computation and dynamic load balancing requirements, an offloaded task often
goes through a multi-stage pipeline that spans across multiple compute nodes
and proxies interconnected via a dedicated network fabric within a given edge
computing facility. As the latest hardware-accelerated transport technologies
such as RDMA and GPUDirect RDMA are adopted to build such network fabric, there
is a need for good understanding of the full potential of these technologies in
the context of computation offload and the effect of different factors such as
GPU scheduling and characteristics of computation on the net performance gain
achievable by these technologies. This paper unveils detailed insights into the
latency overhead in typical machine learning (ML)-based computation pipelines
and analyzes the potential benefits of adopting hardware-accelerated
communication. To this end, we build a model-serving framework that supports
various communication mechanisms. Using the framework, we identify performance
bottlenecks in state-of-the-art model-serving pipelines and show how
hardware-accelerated communication can alleviate them. For example, we show
that GPUDirect RDMA can save 15--50\% of model-serving latency, which amounts
to 70--160 ms.
