  In recent years, emerging storage hardware technologies have focused on
divergent goals: better performance or lower cost-per-bit. Correspondingly,
data systems that employ these technologies are typically optimized either to
be fast (but expensive) or cheap (but slow). We take a different approach: by
architecting a storage engine to natively utilize two tiers of fast and
low-cost storage technologies, we can achieve a Pareto-efficient balance
between performance and cost-per-bit. This paper presents the design and
implementation of PrismDB, a novel key-value store that exploits two extreme
ends of the spectrum of modern NVMe storage technologies (3D XPoint and QLC
NAND) simultaneously. Our key contribution is how to efficiently migrate and
compact data between two different storage tiers. Inspired by the classic
cost-benefit analysis of log cleaning, we develop a new algorithm for
multi-tiered storage compaction that balances the benefit of reclaiming space
for hot objects in fast storage with the cost of compaction I/O in slow
storage. Compared to the standard use of RocksDB on flash in datacenters today,
PrismDB's average throughput on tiered storage is 3.3$\times$ faster and its
read tail latency is 2$\times$ better, using equivalently priced hardware.
