  The growing computational demands posed by increasingly number of neural
network's parameters necessitate low-memory-consumption training approaches.
Previous memory reduction techniques, such as Low-Rank Adaptation (LoRA) and
ReLoRA, suffer from the limitation of low rank and saddle point issues,
particularly during intensive tasks like pre-training. In this paper, we
propose Sparse Spectral Training (SST), an advanced training methodology that
updates all singular values and selectively updates singular vectors of network
weights, thereby optimizing resource usage while closely approximating
full-rank training. SST refines the training process by employing a targeted
updating strategy for singular vectors, which is determined by a multinomial
sampling method weighted by the significance of the singular values, ensuring
both high performance and memory reduction. Through comprehensive testing on
both Euclidean and hyperbolic neural networks across various tasks, including
natural language generation, machine translation, node classification and link
prediction, SST demonstrates its capability to outperform existing memory
reduction training methods and is comparable with full-rank training in some
cases. On OPT-125M, with rank equating to 8.3% of embedding dimension, SST
reduces the perplexity gap to full-rank training by 67.6%, demonstrating a
significant reduction of the performance loss with prevalent low-rank methods.
This approach offers a strong alternative to traditional training techniques,
paving the way for more efficient and scalable neural network training
solutions.
