  Responding to the "datacenter tax" and "killer microseconds" problems for
datacenter applications, diverse solutions including Smart NIC-based ones have
been proposed. Nonetheless, they often suffer from high overhead of
communications over network and/or PCIe links. To tackle the limitations of the
current solutions, this paper proposes ORCA, a holistic network and
architecture co-design solution that leverages current RDMA and emerging
cache-coherent off-chip interconnect technologies. Specifically, ORCA consists
of four hardware and software components: (1) unified abstraction of inter- and
intra-machine communications managed by one-sided RDMA write and cache-coherent
memory write; (2) efficient notification of requests to accelerators assisted
by cache coherence; (3) cache-coherent accelerator architecture directly
processing requests received by NIC; and (4) adaptive device-to-host data
transfer for modern server memory systems consisting of both DRAM and NVM
exploiting state-of-the-art features in CPUs and PCIe. We prototype ORCA with a
commercial system and evaluate three popular datacenter applications: in-memory
key-value store, chain replication-based distributed transaction system, and
deep learning recommendation model inference. The evaluation shows that ORCA
provides 30.1~69.1% lower latency, up to 2.5x higher throughput, and 3x higher
power efficiency than the current state-of-the-art solutions.
