  Combining persistent memory (PM) with RDMA is a promising approach to
performant replicated distributed key-value stores (KVSs). However, existing
replication approaches do not work well when applied to PM KVSs: 1) Using RPC
induces software queueing and execution at backups, increasing request latency;
2) Using one-sided RDMA WRITE causes many streams of small PM writes, leading
to severe device-level write amplification (DLWA) on PM. In this paper, we
propose Rowan, an efficient RDMA abstraction to handle replication writes in PM
KVSs; it aggregates concurrent remote writes from different servers, and lands
these writes to PM in a sequential (thus low DLWA) and one-sided (thus low
latency) manner. We realize Rowan with off-the-shelf RDMA NICs. Further, we
build Rowan-KV, a log-structured PM KVS using Rowan for replication. Evaluation
shows that under write-intensive workloads, compared with PM KVSs using RPC and
RDMA WRITE for replication, Rowan-KV boosts throughput by 1.22X and 1.39X as
well as lowers median PUT latency by 1.77X and 2.11X, respectively, while
largely eliminating DLWA.
