  Attention is all we need as long as we have enough data. Even so, it is
sometimes not easy to determine how much data is enough while the models are
becoming larger and larger. In this paper, we propose HYDRA heads, lightweight
pretrained linguistic self-attention heads to inject knowledge into transformer
models without pretraining them again. Our approach is a balanced paradigm
between leaving the models to learn unsupervised and forcing them to conform to
linguistic knowledge rigidly as suggested in previous studies. Our experiment
proves that the approach is not only the boost performance of the model but
also lightweight and architecture friendly. We empirically verify our framework
on benchmark datasets to show the contribution of linguistic knowledge to a
transformer model. This is a promising result for a new approach to
transferring knowledge from linguistic resources into transformer-based models.
