  Over the past decade, the fourth paradigm of data-intensive science rapidly
became a major driving concept of multiple application domains encompassing and
generating large-scale devices such as light sources and cutting edge
telescopes. The success of data-intensive projects subsequently triggered the
next generation of machine learning approaches. These new artificial
intelligent systems clearly represent a paradigm shift from data processing
pipelines towards the fifth paradigm of composite cognitive applications
requiring the integration of Big Data processing platforms and HPC
technologies. The paper addresses the existing impedance mismatch between
data-intensive and compute-intensive ecosystems by presenting the Spark-MPI
approach based on the MPI Exascale Process Management Interface (PMIx). The
approach is demonstrated within the context of hybrid MPI/GPU ptychographic
image reconstruction pipelines and distributed deep learning applications.
