  The scaling of neural networks with increasing data and model sizes
necessitates more efficient deep learning algorithms. This paper addresses the
memory footprint challenge in neural network training by proposing a
modification to the handling of activation tensors in pointwise nonlinearity
layers. Traditionally, these layers save the entire input tensor for the
backward pass, leading to substantial memory use. Our method involves saving
the output tensor instead, reducing the memory required when the subsequent
layer also saves its input tensor. This approach is particularly beneficial for
transformer-based architectures like GPT, BERT, Mistral, and Llama. Application
of our method involves taken an inverse function of nonlinearity. To the best
of our knowledge, that can not be done analitically and instead we buid an
accurate approximations using simpler functions. Experimental results confirm
that our method significantly reduces memory usage without affecting training
accuracy. The implementation is available at
https://github.com/PgLoLo/optiacts.
