  In the literature, various reversible deep neural networks (DNN) models have
been proposed to reduce memory consumption or improve data-throughput in the
training process. However, almost all existing reversible DNNs either are
constrained to have special structures or are constructed by modifying the
original DNN architectures considerably to enable reversibility. In this work,
we propose exact bit-level reversible transformers without changing the
architectures in the inference procedure. The basic idea is to first treat each
transformer block as the Euler integration approximation for solving an
ordinary differential equation (ODE) and then incorporate the technique of
bidirectional integration approximation (BDIA) (see [26]) for BDIA-based
diffusion inversion) into the neural architecture together with activation
quantization to make it exactly bit-level reversible, referred to as
BDIA-transformer. In the training process, we let a hyper-parameter $\gamma$ in
BDIA-transformer randomly take one of the two values $\{0.5, -0.5\}$ per
transformer block for averaging two consecutive integration approximations,
which regularizes the models for improving the validation accuracy.
Light-weight side information per transformer block is required to be stored in
the forward process to account for binary quantization loss to enable exact
bit-level reversibility. In the inference procedure, the expectation
$\mathbb{E}(\gamma)=0$ is taken to make the resulting architectures of
BDIA-transformer be identical to transformers up to activation quantization.
Empirical study indicates that BDIA-transformers outperform their original
counterparts notably due to the regularization effect of the $\gamma$
parameter.
