  The growing memory demands of modern applications have driven the adoption of
far memory technologies in data centers to provide cost-effective,
high-capacity memory solutions. However, far memory presents new performance
challenges because its access latencies are significantly longer and more
variable than local DRAM. For applications to achieve acceptable performance on
far memory, a high degree of memory-level parallelism (MLP) is needed to
tolerate the long access latency. While modern out-of-order processors are
capable of exploiting a certain degree of MLP, they are constrained by resource
limitations and hardware complexity. The key obstacle is the synchronous memory
access semantics of traditional load/store instructions, which occupy critical
hardware resources for a long time. The longer far memory latencies exacerbate
this limitation.
  This paper proposes a set of Asynchronous Memory Access Instructions (AMI)
and its supporting function unit, Asynchronous Memory Access Unit (AMU), inside
a contemporary Out-of-Order Core. AMI separates memory request issuing from
response handling to reduce resource occupation. Additionally, AMU architecture
supports up to several hundreds of asynchronous memory requests through
re-purposing a portion of L2 Cache as scratchpad memory (SPM) to provide
sufficient temporal storage. Together with a coroutine-based programming
framework, this scheme can achieve significantly higher MLP for hiding far
memory latencies.
  Evaluation with a cycle-accurate simulation shows AMI achieves 2.42x speedup
on average for memory-bound benchmarks with 1us additional far memory latency.
Over 130 outstanding requests are supported with 26.86x speedup for GUPS
(random access) with 5 us latency. These demonstrate how the techniques tackle
far memory performance impacts through explicit MLP expression and latency
adaptation.
