  Modern NVMe SSDs and RDMA networks provide dramatically higher bandwidth and
concurrency. Existing networked storage systems (e.g., NVMe over Fabrics) fail
to fully exploit these new devices due to inefficient storage ordering
guarantees. Severe synchronous execution for storage order in these systems
stalls the CPU and I/O devices and lowers the CPU and I/O performance
efficiency of the storage system.
  We present Rio, a new approach to the storage order of remote storage access.
The key insight in Rio is that the layered design of the software stack, along
with the concurrent and asynchronous network and storage devices, makes the
storage stack conceptually similar to the CPU pipeline. Inspired by the CPU
pipeline that executes out-of-order and commits in-order, Rio introduces the
I/O pipeline that allows internal out-of-order and asynchronous execution for
ordered write requests while offering intact external storage order to
applications. Together with merging consecutive ordered requests, these design
decisions make for write throughput and CPU efficiency close to that of
orderless requests.
  We implement Rio in Linux NVMe over RDMA stack, and further build a file
system named RioFS atop Rio. Evaluations show that Rio outperforms Linux NVMe
over RDMA and a state-of-the-art storage stack named Horae by two orders of
magnitude and 4.9 times on average in terms of throughput of ordered write
requests, respectively. RioFS increases the throughput of RocksDB by 1.9 times
and 1.5 times on average, against Ext4 and HoraeFS, respectively.
