  Distributional semantics based on neural approaches is a cornerstone of
Natural Language Processing, with surprising connections to human meaning
representation as well. Recent Transformer-based Language Models have proven
capable of producing contextual word representations that reliably convey
sense-specific information, simply as a product of self-supervision. Prior work
has shown that these contextual representations can be used to accurately
represent large sense inventories as sense embeddings, to the extent that a
distance-based solution to Word Sense Disambiguation (WSD) tasks outperforms
models trained specifically for the task. Still, there remains much to
understand on how to use these Neural Language Models (NLMs) to produce sense
embeddings that can better harness each NLM's meaning representation abilities.
In this work we introduce a more principled approach to leverage information
from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We
also emphasize the versatility of these sense embeddings in contrast to
task-specific models, applying them on several sense-related tasks, besides
WSD, while demonstrating improved performance using our proposed approach over
prior work focused on sense embeddings. Finally, we discuss unexpected findings
regarding layer and model performance variations, and potential applications
for downstream tasks.
