  Software vendors often report performance numbers for the sweet spot or
running on specialized hardware with specific workload parameters and without
realistic failures. Accurate benchmarks at the persistence layer are crucial,
as failures may cause unrecoverable errors such as data loss, inconsistency or
corruption. To accurately evaluate data stores and other microservices at
Netflix, we developed Netflix Data Benchmark (NDBench), a Cloud benchmark tool.
It can be deployed in a loosely-coupled fashion with the ability to dynamically
change the benchmark parameters at runtime so we can rapidly iterate on
different tests and failure modes. NDBench offers pluggable patterns and loads,
support for pluggable client APIs, and was designed to run continually. This
design enabled us to test long-running maintenance jobs that may affect the
performance, test numerous different systems under adverse conditions, and
uncover long-term issues like memory leaks or heap pressure.
