  Apache Hadoop and Spark are gaining prominence in Big Data processing and
analytics. Both of them are widely deployed on Internet companies. On the other
hand, high-performance data analysis requirements are causing academical and
industrial communities to adopt state-of-the-art technologies in HPC to solve
Big Data problems. Recently, we have proposed a key-value pair based
communication library, DataMPI, which is extending MPI to support
Hadoop/Spark-like Big Data Computing jobs. In this paper, we use BigDataBench,
a Big Data benchmark suite, to do comprehensive studies on performance and
resource utilization characterizations of Hadoop, Spark and DataMPI. From our
experiments, we observe that the job execution time of DataMPI has up to 55%
and 39% speedups compared with those of Hadoop and Spark, respectively. Most of
the benefits come from the high-efficiency communication mechanisms in DataMPI.
We also notice that the resource (CPU, memory, disk and network I/O)
utilizations of DataMPI are also more efficient than those of the other two
frameworks.
