  Spark is an in-memory analytics platform that targets commodity server
environments today. It relies on the Hadoop Distributed File System (HDFS) to
persist intermediate checkpoint states and final processing results. In Spark,
immutable data are used for storing data updates in each iteration, making it
inefficient for long running, iterative workloads. A non-deterministic garbage
collector further worsens this problem. Sparkle is a library that optimizes
memory usage in Spark. It exploits large shared memory to achieve better data
shuffling and intermediate storage. Sparkle replaces the current TCP/IP-based
shuffle with a shared memory approach and proposes an off-heap memory store for
efficient updates. We performed a series of experiments on scale-out clusters
and scale-up machines. The optimized shuffle engine leveraging shared memory
provides 1.3x to 6x faster performance relative to Vanilla Spark. The off-heap
memory store along with the shared-memory shuffle engine provides more than 20x
performance increase on a probabilistic graph processing workload that uses a
large-scale real-world hyperlink graph. While Sparkle benefits at most from
running on large memory machines, it also achieves 1.6x to 5x performance
improvements over scale out cluster with equivalent hardware setting.
