  Even in highly-developed countries, as many as 15-30\% of the population can
only understand texts written using a basic vocabulary. Their understanding of
everyday texts is limited, which prevents them from taking an active role in
society and making informed decisions regarding healthcare, legal
representation, or democratic choice. Lexical simplification is a natural
language processing task that aims to make text understandable to everyone by
replacing complex vocabulary and expressions with simpler ones, while
preserving the original meaning. It has attracted considerable attention in the
last 20 years, and fully automatic lexical simplification systems have been
proposed for various languages. The main obstacle for the progress of the field
is the absence of high-quality datasets for building and evaluating lexical
simplification systems. We present a new benchmark dataset for lexical
simplification in English, Spanish, and (Brazilian) Portuguese, and provide
details about data selection and annotation procedures. This is the first
dataset that offers a direct comparison of lexical simplification systems for
three languages. To showcase the usability of the dataset, we adapt two
state-of-the-art lexical simplification systems with differing architectures
(neural vs.\ non-neural) to all three languages (English, Spanish, and
Brazilian Portuguese) and evaluate their performances on our new dataset. For a
fairer comparison, we use several evaluation measures which capture varied
aspects of the systems' efficacy, and discuss their strengths and weaknesses.
We find a state-of-the-art neural lexical simplification system outperforms a
state-of-the-art non-neural lexical simplification system in all three
languages. More importantly, we find that the state-of-the-art neural lexical
simplification systems perform significantly better for English than for
Spanish and Portuguese.
