We provided many resources of parallel programming tutorials. The following are the ones we have linked so far.
Basic parallel computing tutorial
Cell processor programming
Intel Threading Building Blocks
In Part 7, we visit Message Passing Interface (MPI), the de facto standard for writing parallel programs running on a distributed memory system, such as a compute cluster, and is widely implemented.
Our choice for basic introductory MPI tutorial is again from Blaise Barney’s collection of tutorials. An excerpt from this tutorial:
“The Message Passing Interface Standard (MPI) is a message passing library standard based on the consensus of the MPI Forum, which has over 40 participating organizations, including vendors, researchers, software library developers, and users. The goal of the Message Passing Interface is to establish a portable, efficient, and flexible standard for message passing that will be widely used for writing message passing programs. As such, MPI is the first standardized, vendor independent, message passing library. The advantages of developing message passing software using MPI closely match the design goals of portability, efficiency, and flexibility.”
MCS Divison of Argonne National Lab provides a page with many links to MPI tutorials here. Some of these tutorials have example code available for downloading.
Some More Interesting Sources
Tutorial on MPI: The Message-Passing Interface by William Gropp
Getting started with LAM/MPI
NERSC MPI Tutorial
Basic MPI Tutorial
Adaptive Message Passing Interface (AMPI) Tutorial [ppt slides] [AMPI]
Open MPI Website
Fault Tolerant MPI
Message Passing Interface Forum
MPI 3.0 Standardization Effort
MPI Specification Documents
The Evolution of MPI by Bill Gropp
* Gropp, William; Lusk, Ewing; Skjellum, Anthony. (1999) Using MPI, 2nd Edition: portable Parallel Programming with the Message Passing Interface. MIT Press In Scientific And Engineering Computation Series, Cambridge, MA, USA. 395 pp. ISBN 978-0-262-57132-6
* Gropp, William; R Thakur, E Lusk (1999) Using MPI-2: Advanced Features of the Message Passing Interface - MIT Press Cambridge, MA, USA ISBN 0-262-57133-1
* Pacheco, Peter S. (1997) Parallel Programming with MPI. 500 pp. Morgan Kaufmann ISBN 1558603395