  State of the art natural language processing tools are built on
context-dependent word embeddings, but no direct method for evaluating these
representations currently exists. Standard tasks and datasets for intrinsic
evaluation of embeddings are based on judgements of similarity, but ignore
context; standard tasks for word sense disambiguation take account of context
but do not provide continuous measures of meaning similarity. This paper
describes an effort to build a new dataset, CoSimLex, intended to fill this
gap. Building on the standard pairwise similarity task of SimLex-999, it
provides context-dependent similarity measures; covers not only discrete
differences in word sense but more subtle, graded changes in meaning; and
covers not only a well-resourced language (English) but a number of
less-resourced languages. We define the task and evaluation metrics, outline
the dataset collection methodology, and describe the status of the dataset so
far.
