  Split and Rephrase is a text simplification task of rewriting a complex
sentence into simpler ones. As a relatively new task, it is paramount to ensure
the soundness of its evaluation benchmark and metric. We find that the widely
used benchmark dataset universally contains easily exploitable syntactic cues
caused by its automatic generation process. Taking advantage of such cues, we
show that even a simple rule-based model can perform on par with the
state-of-the-art model. To remedy such limitations, we collect and release two
crowdsourced benchmark datasets. We not only make sure that they contain
significantly more diverse syntax, but also carefully control for their quality
according to a well-defined set of criteria. While no satisfactory automatic
metric exists, we apply fine-grained manual evaluation based on these criteria
using crowdsourcing, showing that our datasets better represent the task and
are significantly more challenging for the models.
