  How much information do NLP tasks really need from a transformer's attention
mechanism at application-time (inference)? From recent work, we know that there
is sparsity in transformers and that the floating-points within its computation
can be discretized to fewer values with minimal loss to task accuracies.
However, this requires retraining or even creating entirely new models, both of
which can be expensive and carbon-emitting. Focused on optimizations that do
not require training, we systematically study the full range of typical
attention values necessary. This informs the design of an inference-time
quantization technique using both pruning and log-scaled mapping which produces
only a few (e.g. $2^3$) unique values. Over the tasks of question answering and
sentiment analysis, we find nearly 80% of attention values can be pruned to
zeros with minimal ($< 1.0\%$) relative loss in accuracy. We use this pruning
technique in conjunction with quantizing the attention values to only a 3-bit
format, without retraining, resulting in only a 0.8% accuracy reduction on
question answering with fine-tuned RoBERTa.
