  Identifying words that impact a task's performance more than others is a
challenge in natural language processing. Transformers models have recently
addressed this issue by incorporating an attention mechanism that assigns
greater attention (i.e., relevance) scores to some words than others. Because
of the attention mechanism's high computational cost, transformer models
usually have an input-length limitation caused by hardware constraints. This
limitation applies to many transformers, including the well-known bidirectional
encoder representations of the transformer (BERT) model. In this paper, we
examined BERT's attention assignment mechanism, focusing on two questions: (1)
How can attention be employed to reduce input length? (2) How can attention be
used as a control mechanism for conditional text generation? We investigated
these questions in the context of a text classification task. We discovered
that BERT's early layers assign more critical attention scores for text
classification tasks compared to later layers. We demonstrated that the first
layer's attention sums could be used to filter tokens in a given sequence,
considerably decreasing the input length while maintaining good test accuracy.
We also applied filtering, which uses a compute-efficient semantic similarities
algorithm, and discovered that retaining approximately 6\% of the original
sequence is sufficient to obtain 86.5\% accuracy. Finally, we showed that we
could generate data in a stable manner and indistinguishable from the original
one by only using a small percentage (10\%) of the tokens with high attention
scores according to BERT's first layer.
