  Artificial neural networks with millions of adjustable parameters and a
similar number of training examples are a potential solution for difficult,
large-scale pattern recognition problems in areas such as speech and face
recognition, classification of large volumes of web data, and finance. The
bottleneck is that neural network training involves iterative gradient descent
and is extremely computationally intensive. In this paper we present a
technique for distributed training of Ultra Large Scale Neural Networks (ULSNN)
on Bunyip, a Linux-based cluster of 196 Pentium III processors. To illustrate
ULSNN training we describe an experiment in which a neural network with 1.73
million adjustable parameters was trained to recognize machine-printed Japanese
characters from a database containing 9 million training patterns. The training
runs with a average performance of 163.3 GFlops/s (single precision). With a
machine cost of \$150,913, this yields a price/performance ratio of
92.4c/MFlops/s (single precision). For comparison purposes, training using
double precision and the ATLAS DGEMM produces a sustained performance of 70
MFlops/s or \$2.16 / MFlop/s (double precision).
