  Transformer-based language models have taken many fields in NLP by storm.
BERT and its derivatives dominate most of the existing evaluation benchmarks,
including those for Word Sense Disambiguation (WSD), thanks to their ability in
capturing context-sensitive semantic nuances. However, there is still little
knowledge about their capabilities and potential limitations in encoding and
recovering word senses. In this article, we provide an in-depth quantitative
and qualitative analysis of the celebrated BERT model with respect to lexical
ambiguity. One of the main conclusions of our analysis is that BERT can
accurately capture high-level sense distinctions, even when a limited number of
examples is available for each word sense. Our analysis also reveals that in
some cases language models come close to solving coarse-grained noun
disambiguation under ideal conditions in terms of availability of training data
and computing resources. However, this scenario rarely occurs in real-world
settings and, hence, many practical challenges remain even in the
coarse-grained setting. We also perform an in-depth comparison of the two main
language model based WSD strategies, i.e., fine-tuning and feature extraction,
finding that the latter approach is more robust with respect to sense bias and
it can better exploit limited available training data. In fact, the simple
feature extraction strategy of averaging contextualized embeddings proves
robust even using only three training sentences per word sense, with minimal
improvements obtained by increasing the size of this training data.
