  Contextualized embeddings such as BERT can serve as strong input
representations to NLP tasks, outperforming their static embeddings
counterparts such as skip-gram, CBOW and GloVe. However, such embeddings are
dynamic, calculated according to a sentence-level context, which limits their
use in lexical semantics tasks. We address this issue by making use of dynamic
embeddings as word representations in training static embeddings, thereby
leveraging their strong representation power for disambiguating context
information. Results show that this method leads to improvements over
traditional static embeddings on a range of lexical semantics tasks, obtaining
the best reported results on seven datasets.
