  Automatic text generation based on neural language models has achieved
performance levels that make the generated text almost indistinguishable from
those written by humans. Despite the value that text generation can have in
various applications, it can also be employed for malicious tasks. The
diffusion of such practices represent a threat to the quality of academic
publishing. To address these problems, we propose in this paper two datasets
comprised of artificially generated research content: a completely synthetic
dataset and a partial text substitution dataset. In the first case, the content
is completely generated by the GPT-2 model after a short prompt extracted from
original papers. The partial or hybrid dataset is created by replacing several
sentences of abstracts with sentences that are generated by the Arxiv-NLP
model. We evaluate the quality of the datasets comparing the generated texts to
aligned original texts using fluency metrics such as BLEU and ROUGE. The more
natural the artificial texts seem, the more difficult they are to detect and
the better is the benchmark. We also evaluate the difficulty of the task of
distinguishing original from generated text by using state-of-the-art
classification models.
