  Big data analytics requires high programmer productivity and high performance
simultaneously on large-scale clusters. However, current big data analytics
frameworks (e.g. Apache Spark) have prohibitive runtime overheads since they
are library-based. We introduce a novel auto-parallelizing compiler approach
that exploits the characteristics of the data analytics domain such as the
map/reduce parallel pattern and is robust, unlike previous auto-parallelization
methods. Using this approach, we build High Performance Analytics Toolkit
(HPAT), which parallelizes high-level scripting (Julia) programs automatically,
generates efficient MPI/C++ code, and provides resiliency. Furthermore, it
provides automatic optimizations for scripting programs, such as fusion of
array operations. Thus, HPAT is 369x to 2033x faster than Spark on the Cori
supercomputer and 20x to 256x times on Amazon AWS.
