  The multi-head self-attention mechanism of the transformer model has been
thoroughly investigated recently. In one vein of study, researchers are
interested in understanding why and how transformers work. In another vein,
researchers propose new attention augmentation methods to make transformers
more accurate, efficient and interpretable. In this paper, we combine these two
lines of research in a human-in-the-loop pipeline to first discover important
task-specific attention patterns. Then those patterns are injected, not only to
smaller models, but also to the original model. The benefits of our pipeline
and discovered patterns are demonstrated in two case studies with extractive
summarization and topic segmentation. After discovering interpretable patterns
in BERT-based models fine-tuned for the two downstream tasks, experiments
indicate that when we inject the patterns into attention heads, the models show
considerable improvements in accuracy and efficiency.
