  Distributed data processing frameworks (e.g., Hadoop, Spark, and Flink) are
widely used to distribute data among computing nodes of a cloud. Recently,
there have been increasing efforts aimed at evaluating the performance of
distributed data processing frameworks hosted in private and public clouds.
However, there is a paucity of research on evaluating the performance of these
frameworks hosted in a hybrid cloud, which is an emerging cloud model that
integrates private and public clouds to use the best of both worlds. Therefore,
in this paper, we evaluate the performance of Hadoop, Spark, and Flink in a
hybrid cloud in terms of execution time, resource utilization, horizontal
scalability, vertical scalability, and cost. For this study, our hybrid cloud
consists of OpenStack (private cloud) and MS Azure (public cloud). We use both
batch and iterative workloads for the evaluation. Our results show that in a
hybrid cloud (i) the execution time increases as more nodes are borrowed by the
private cloud from the public cloud, (ii) Flink outperforms Spark, which in
turn outperforms Hadoop in terms of execution time, (iii) Hadoop transfers the
largest amount of data among the nodes during the workload execution while
Spark transfers the least amount of data, (iv) all three frameworks
horizontally scale better as compared to vertical scaling, and (v) Spark is
found to be least expensive in terms of $ cost for data processing while Hadoop
is found the most expensive.
