  Krylov methods are a key way of solving large sparse linear systems of
equations, but suffer from poor strong scalabilty on distributed memory
machines. This is due to high synchronization costs from large numbers of
collective communication calls alongside a low computational workload. Enlarged
Krylov methods address this issue by decreasing the total iterations to
convergence, an artifact of splitting the initial residual and resulting in
operations on block vectors. In this paper, we present a performance study of
an Enlarged Krylov Method, Enlarged Conjugate Gradients (ECG), noting the
impact of block vectors on parallel performance at scale. Most notably, we
observe the increased overhead of point-to-point communication as a result of
denser messages in the sparse matrix-block vector multiplication kernel.
Additionally, we present models to analyze expected performance of ECG, as well
as, motivate design decisions. Most importantly, we introduce a new
point-to-point communication approach based on node-aware communication
techniques that increases efficiency of the method at scale.
