  By design, word embeddings are unable to model the dynamic nature of words'
semantics, i.e., the property of words to correspond to potentially different
meanings. To address this limitation, dozens of specialized meaning
representation techniques such as sense or contextualized embeddings have been
proposed. However, despite the popularity of research on this topic, very few
evaluation benchmarks exist that specifically focus on the dynamic semantics of
words. In this paper we show that existing models have surpassed the
performance ceiling of the standard evaluation dataset for the purpose, i.e.,
Stanford Contextual Word Similarity, and highlight its shortcomings. To address
the lack of a suitable benchmark, we put forward a large-scale Word in Context
dataset, called WiC, based on annotations curated by experts, for generic
evaluation of context-sensitive representations. WiC is released in
https://pilehvar.github.io/wic/.
