  Deep learning often faces the challenge of efficiently processing dynamic
inputs, such as sensor data or user inputs. For example, an AI writing
assistant is required to update its suggestions in real time as a document is
edited. Re-running the model each time is expensive, even with compression
techniques like knowledge distillation, pruning, or quantization. Instead, we
take an incremental computing approach, looking to reuse calculations as the
inputs change. However, the dense connectivity of conventional architectures
poses a major obstacle to incremental computation, as even minor input changes
cascade through the network and restrict information reuse. To address this, we
use vector quantization to discretize intermediate values in the network, which
filters out noisy and unnecessary modifications to hidden neurons, facilitating
the reuse of their values. We apply this approach to the transformers
architecture, creating an efficient incremental inference algorithm with
complexity proportional to the fraction of the modified inputs. Our experiments
with adapting the OPT-125M pre-trained language model demonstrate comparable
accuracy on document classification while requiring 12.1X (median) fewer
operations for processing sequences of atomic edits.
