  We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense
Disambiguation. More specifically, we introduce a framework for Target Sense
Verification of Words in Context which grounds its uniqueness in the
formulation as a binary classification task thus being independent of external
sense inventories, and the coverage of various domains. This makes the dataset
highly flexible for the evaluation of a diverse set of models and systems in
and across domains. WiC-TSV provides three different evaluation settings,
depending on the input signals provided to the model. We set baseline
performance on the dataset using state-of-the-art language models. Experimental
results show that even though these models can perform decently on the task,
there remains a gap between machine and human performance, especially in
out-of-domain settings. WiC-TSV data is available at
https://competitions.codalab.org/competitions/23683
