  Resource disaggregation offers a cost effective solution to resource scaling,
utilization, and failure-handling in data centers by physically separating
hardware devices in a server. Servers are architected as pools of processor,
memory, and storage devices, organized as independent failure-isolated
components interconnected by a high-bandwidth network. A critical challenge,
however, is the high performance penalty of accessing data from a remote memory
module over the network. Addressing this challenge is difficult as
disaggregated systems have high runtime variability in network
latencies/bandwidth, and page migration can significantly delay critical path
cache line accesses in other pages.
  This paper conducts a characterization analysis on different data movement
strategies in fully disaggregated systems, evaluates their performance
overheads in a variety of workloads, and introduces DaeMon, the first
software-transparent mechanism to significantly alleviate data movement
overheads in fully disaggregated systems. First, to enable scalability to
multiple hardware components in the system, we enhance each compute and memory
unit with specialized engines that transparently handle data migrations.
Second, to achieve high performance and provide robustness across various
network, architecture and application characteristics, we implement a
synergistic approach of bandwidth partitioning, link compression, decoupled
data movement of multiple granularities, and adaptive granularity selection in
data movements. We evaluate DaeMon in a wide variety of workloads at different
network and architecture configurations using a state-of-the-art accurate
simulator. DaeMon improves system performance and data access costs by
2.39$\times$ and 3.06$\times$, respectively, over the widely-adopted approach
of moving data at page granularity.
