  Transformers are becoming increasingly popular due to their superior
performance over conventional convolutional neural networks(CNNs). However,
transformers usually require a much larger amount of memory to train than CNNs,
which prevents their application in many low resource settings. Local learning,
which divides the network into several distinct modules and trains them
individually, is a promising alternative to the end-to-end (E2E) training
approach to reduce the amount of memory for training and to increase
parallelism. This paper is the first to apply Local Learning on transformers
for this purpose. The standard CNN-based local learning method, InfoPro [32],
reconstructs the input images for each module in a CNN. However, reconstructing
the entire image does not generalize well. In this paper, we propose a new
mechanism for each local module, where instead of reconstructing the entire
image, we reconstruct its input features, generated from previous modules. We
evaluate our approach on 4 commonly used datasets and 3 commonly used decoder
structures on Swin-Tiny. The experiments show that our approach outperforms
InfoPro-Transformer, the InfoPro with Transfomer backbone we introduced, by at
up to 0.58% on CIFAR-10, CIFAR-100, STL-10 and SVHN datasets, while using up to
12% less memory. Compared to the E2E approach, we require 36% less GPU memory
when the network is divided into 2 modules and 45% less GPU memory when the
network is divided into 4 modules.
