  Fine-tuning pretrained large models to downstream tasks is an important
problem, which however suffers from huge memory overhead due to large-scale
parameters. This work strives to reduce memory overhead in fine-tuning from
perspectives of activation function and layer normalization. To this end, we
propose the Approximate Backpropagation (Approx-BP) theory, which provides the
theoretical feasibility of decoupling the forward and backward passes. We apply
our Approx-BP theory to backpropagation training and derive memory-efficient
alternatives of GELU and SiLU activation functions, which use derivative
functions of ReLUs in the backward pass while keeping their forward pass
unchanged. In addition, we introduce a Memory-Sharing Backpropagation strategy,
which enables the activation memory to be shared by two adjacent layers,
thereby removing activation memory usage redundancy. Our method neither induces
extra computation nor reduces training efficiency. We conduct extensive
experiments with pretrained vision and language models, and the results
demonstrate that our proposal can reduce up to $\sim$$30\%$ of the peak memory
usage. Our code is released at https://github.com/yyyyychen/LowMemoryBP.
