  Existing approaches to train neural networks that use large images require to
either crop or down-sample data during pre-processing, use small batch sizes,
or split the model across devices mainly due to the prohibitively limited
memory capacity available on GPUs and emerging accelerators. These techniques
often lead to longer time to convergence or time to train (TTT), and in some
cases, lower model accuracy. CPUs, on the other hand, can leverage significant
amounts of memory. While much work has been done on parallelizing neural
network training on multiple CPUs, little attention has been given to tune
neural network training with large images on CPUs. In this work, we train a
multi-scale convolutional neural network (M-CNN) to classify large biomedical
images for high content screening in one hour. The ability to leverage large
memory capacity on CPUs enables us to scale to larger batch sizes without
having to crop or down-sample the input images. In conjunction with large batch
sizes, we find a generalized methodology of linearly scaling of learning rate
and train M-CNN to state-of-the-art (SOTA) accuracy of 99% within one hour. We
achieve fast time to convergence using 128 two socket Intel Xeon 6148 processor
nodes with 192GB DDR4 memory connected with 100Gbps Intel Omnipath
architecture.
