  The sparse matrix-vector multiply (SpMV) operation is a key computational
kernel in many simulations and linear solvers. The large communication
requirements associated with a reference implementation of a parallel SpMV
result in poor parallel scalability. The cost of communication depends on the
physical locations of the send and receive processes: messages injected into
the network are more costly than messages sent between processes on the same
node. In this paper, a node aware parallel SpMV (NAPSpMV) is introduced to
exploit knowledge of the system topology, specifically the node-processor
layout, to reduce costs associated with communication. The values of the input
vector are redistributed to minimize both the number and the size of messages
that are injected into the network during a SpMV, leading to a reduction in
communication costs. A variety of computational experiments that highlight the
efficiency of this approach are presented.
