  Scale-out parallel processing based on MPI is a 25-year-old standard with at
least another decade of preceding history of enabling technologies in the High
Performance Computing community. Newer frameworks such as MapReduce, Hadoop,
and Spark represent industrial scalable computing solutions that have received
broad adoption because of their comparative simplicity of use, applicability to
relevant problems, and ability to harness scalable, distributed resources.
While MPI provides performance and portability, it lacks in productivity and
fault tolerance. Likewise, Spark is a specific example of a current-generation
MapReduce and data-parallel computing infrastructure that addresses those goals
but in turn lacks peer communication support to allow featherweight, highly
scalable peer-to-peer data-parallel code sections. The key contribution of this
paper is to demonstrate how to introduce the collective and point-to-point peer
communication concepts of MPI into a Spark environment. This is done in order
to produce performance-portable, peer-oriented and group-oriented communication
services while retaining the essential, desirable properties of Spark.
Additional concepts of fault tolerance and productivity are considered. This
approach is offered in contrast to adding MapReduce framework as
upper-middleware based on a traditional MPI implementation as baseline
infrastructure.
