A ``cluster'' is a collection of complete computers (nodes) that are physically interconnected by a high-performance ``local area network'' (LAN). Typically each node is a workstation or personal computer (PC). Clusters permit running parallel jobs using PVM or MPI implemented over the network as well as permitting independent use of the nodes for task farming.
The advantages of cluster computing derive from the fact that off-the-shelf commodity components are used. This offers a cost-effective solution to medium-scale computing requirements.
For a good introduction to the concepts of cluster computing and architectures available see ``Scalable Parallel Computing'' by K. Hwang and Z. Xu (WCB/McGraw-Hill, 1998, ISBN 0-07-031798-4).
Cluster computing solutions in the USA are becoming mainstream and may become the dominant system of the future for computational science. A 64x2-way Alpha-based system built by Alta Technology and installed at the University of New Mexico has been accepted into the USA Alliance computational meta-computing grid. A similar system, the CPlant
http://www.cs.sandia.gov/cplant, is supported by Compaq under a 4-year agreement with the US DoE and is now being used at Sandia National Laboratory with a Myrinet high-performance switch for enhanced communications. This system forms part of the Accelerated Strategic Computing Initiative (ASCI) Path Forward programme.
One of the first projects, the Beowulf
http://www.beowulf.org, was started at NASA in 1994. This Web page also contains links to many related sites worldwide. Commodity cluster systems are now often known as Beowulf-class computers.
It is important for UK scientists to be able to evaluate this kind of equipment for parallel computing, as has been noted by EPSRC in recent surveys. Daresbury Laboratory therefore, as part of the Distributed Computing (DisCo) Programme, has built a 32-processor Beowulf cluster using 450 MHz Pentium III processors. Currently the processors, which are in the form of off-the-shelf PCs, each with memory and disk but no keyboard or monitor, are connected by dual fast Ethernet switches - 2x Extreme Summit48, one network for IP traffic (e.g. nfs) and the other for MPI message passing. Additional 8-port KVM switches are used to attach a keyboard and monitor to any one of the nodes for administrative purposes. The whole cluster has a single master node (with a backup spare) for compilation and resource management. All nodes are currently running RedHat Linux v6.0.
Applications, such as GAMESS-UK, DL_POLY, ANGUS, CRYSTAL, POL-ERSEM, REALC and CASTEP are being ported to the system for evaluation. Results showing their performance will be posted on the DisCo Web site
http://www.cse.clrc.ac.uk/Activity/DisCo as they become available.
Over the coming months we also plan to evaluate a variety of networking and software options for the system. Some of the options are summarised below. Prices vary, as does performance and robustness, and it is not yet clear what will be the preferred solution for building a large-scale compute server.
Gigabit Ethernet †
QSW QsNet ‡
† Gamma project with Packet Engines NIC.
‡ MPI short message protocol.
Figures in the table are subject to confirmation and depend on what driver hardware and software is used.
Message passing options include implementations of MPI and PVM, but there are others too.
http://www-unix.mcs.anl.gov/mpi/mpich/index.html - Argonne National Laboratory's implementation of MPI
http://www.mpi.nd.edu/lam - Local Area Multicomputer MPI, developed at the Ohio Supercomputer Center and Univ. of Notre Dame
Widely-used compilers include the Gnu family, Portland Group, KAI, Fujitsu, Absoft, NAG etc.. Compaq is about to beta test AlphaLinux compilers which are reputedly excellent. Some people already compile their applications under Digital Unix and run them on Alpha Linux, although this is not permitted under the license conditions.
http://www.absoft.com - FORTRAN77 (f77) and Fortran 90 (f90)
The Portland Group
http://www.pgroup.com (PGI) - High Performance Fortran (pghpf), FORTRAN77 (pgf77), C and C++ (pgcc)
http://www.cs.utk.edu/~ghenry/distrib/archive.htm - BLAS, fast-Fourier transform, hardware performance-monitoring utilities, extended-precision and maths primitives are all available free under restricted licenses
Fast Maths library
http://www.lsc-group.phys.uwm.edu/~www/docs/beowulf/os_updates/fastMath.html and Free Fast Maths library - makes standard mathematical functions much faster
http://www.nag.co.uk Parallel Library - a version tuned for Beowulf systems is available commercially
http://www.beowulf.org/software/bproc.html - making processes visible across nodes, allowing fork()s to happen across nodes, allowing process migration, allowing kill()s to work across nodes, currently pre-alpha release
Cluster patches for procps
http://www.sc.cs.tu-bs.de/pare/results/procps.html - lets you compile /proc-based programs like ps so they report on all processes on the cluster, not just the ones on the machine you're logged into
http://smile.cpe.ku.ac.th/software/scms/index.html Cluster Management System - Run commands on all nodes, shut down individual nodes and sets of nodes, monitor health of nodes. Makes clusters easier to administer.
Parallel Virtual Filesystem
http://ece.clemson.edu/parl/pvfs - LD_PRELOAD-based filesystem modification to let you transparently stripe big files across many disks. Allows high-performance access to big datasets.
Scripts for configuring 'clone' worker nodes
ftp://ftp.sci.usq.edu.au/pub/jacek/beowulf-utils/disk-less - makes adding nodes to a Beowulf painless
ftp://ftp.sci.usq.edu.au/pub/jacek/beowulf-utils/misc_scripts for doing various things on a cluster, backups, shutdowns, reboots, running a command on every node
http://www.par-tec.com supports the ParaStation project and sells clusters and services
See also the NASA Web site mentioned above.
As an example of our initial experiences using the Daresbury Beowulf we show the performance obtained from DL_POLY. The test cases were NaCl MTS Ewald 27000 ions, NaK disilicate glass 8640 ions and Gramicidin in water, SHAKE, 13390 atoms. Results from the 450 MHz Pentium III system are compared with an earlier 260 MHz Pentium II system and Cray T3Es. Clearly the single-node performance of the Pentium III is good compared to the Cray T3E-1200E, but the latter offers superior scalability for parallel programs needing a large number of processors.