  Millions of sensors, mobile applications and machines now generate billions
of events. Specialized many-core key-value stores (KVSs) can ingest and index
these events at high rates (over 100 Mops/s on one machine) if events are
generated on the same machine; however, to be practical and cost-effective they
must ingest events over the network and scale across cloud resources
elastically.
  We present Shadowfax, a new distributed KVS based on FASTER, that
transparently spans DRAM, SSDs, and cloud blob storage while serving 130
Mops/s/VM over commodity Azure VMs using conventional Linux TCP. Beyond high
single-VM performance, Shadowfax uses a unique approach to distributed
reconfiguration that avoids any server-side key ownership checks or cross-core
coordination both during normal operation and migration. Hence, Shadowfax can
shift load in 17 s to improve system throughput by 10 Mops/s with little
disruption. Compared to the state-of-the-art, it has 8x better throughput (than
Seastar+memcached) and avoids costly I/O to move cold data during migration. On
12 machines, Shadowfax retains its high throughput to perform 930 Mops/s,
which, to the best of our knowledge, is the highest reported throughput for a
distributed KVS used for large-scale data ingestion and indexing.
