  Nowadays, several software systems rely on stream processing architectures to
deliver scalable performance and handle large volumes of data in near
real-time. Stream processing frameworks facilitate scalable computing by
distributing the application's execution across multiple machines. Despite
performance being extensively studied, the measurement of fault tolerance-a key
feature offered by stream processing frameworks-has still not been measured
properly with updated and comprehensive testbeds. Moreover, the impact that
fault recovery can have on performance is mostly ignored. This paper provides a
comprehensive analysis of fault recovery performance, stability, and recovery
time in a cloud-native environment with modern open-source frameworks, namely
Flink, Kafka Streams, and Spark Structured Streaming. Our benchmarking analysis
is inspired by chaos engineering to inject failures. Generally, our results
indicate that much has changed compared to previous studies on fault recovery
in distributed stream processing. In particular, the results indicate that
Flink is the most stable and has one of the best fault recovery. Moreover,
Kafka Streams shows performance instabilities after failures, which is due to
its current rebalancing strategy that can be suboptimal in terms of load
balancing. Spark Structured Streaming shows suitable fault recovery performance
and stability, but with higher event latency. Our study intends to (i) help
industry practitioners in choosing the most suitable stream processing
framework for efficient and reliable executions of data-intensive applications;
(ii) support researchers in applying and extending our research method as well
as our benchmark; (iii) identify, prevent, and assist in solving potential
issues in production deployments.
