  The advent of multi-/many-core processors in clusters advocates hybrid
parallel programming, which combines Message Passing Interface (MPI) for
inter-node parallelism with a shared memory model for on-node parallelism.
Compared to the traditional hybrid approach of MPI plus OpenMP, a new, but
promising hybrid approach of MPI plus MPI-3 shared-memory extensions (MPI+MPI)
is gaining attraction. We describe an algorithmic approach for collective
operations (with allgather and broadcast as concrete examples) in the context
of hybrid MPI+MPI, so as to minimize memory consumption and memory copies. With
this approach, only one memory copy is maintained and shared by on-node
processes. This allows the removal of unnecessary on-node copies of replicated
data that are required between MPI processes when the collectives are invoked
in the context of pure MPI. We compare our approach of collectives for hybrid
MPI+MPI and the traditional one for pure MPI, and also have a discussion on the
synchronization that is required to guarantee data integrity. The performance
of our approach has been validated on a Cray XC40 system (Cray MPI) and NEC
cluster (OpenMPI), showing that it achieves comparable or better performance
for allgather operations. We have further validated our approach with a
standard computational kernel, namely distributed matrix multiplication, and a
Bayesian Probabilistic Matrix Factorization code.
