  Identifying words which may cause difficulty for a reader is an essential
step in most lexical text simplification systems prior to lexical substitution
and can also be used for assessing the readability of a text. This task is
commonly referred to as Complex Word Identification (CWI) and is often modelled
as a supervised classification problem. For training such systems, annotated
datasets in which words and sometimes multi-word expressions are labelled
regarding complexity are required. In this paper we analyze previous work
carried out in this task and investigate the properties of CWI datasets for
English. We develop a protocol for the annotation of lexical complexity and use
this to annotate a new dataset, CompLex 2.0. We present experiments using both
new and old datasets to investigate the nature of lexical complexity. We found
that a Likert-scale annotation protocol provides an objective setting that is
superior for identifying the complexity of words compared to a binary
annotation protocol. We release a new dataset using our new protocol to promote
the task of Lexical Complexity Prediction.
