  Approaches to machine generated text detection tend to focus on binary
classification of human versus machine written text. In the scientific domain
where publishers might use these models to examine manuscripts under
submission, misclassification has the potential to cause harm to authors.
Additionally, authors may appropriately use text generation models such as with
the use of assistive technologies like translation tools. In this setting, a
binary classification scheme might be used to flag appropriate uses of
assistive text generation technology as simply machine generated which is a
cause of concern. In our work, we simulate this scenario by presenting a
state-of-the-art detector trained on the DAGPap22 with machine translated
passages from Scielo and find that the model performs at random. Given this
finding, we develop a framework for dataset development that provides a nuanced
approach to detecting machine generated text by having labels for the type of
technology used such as for translation or paraphrase resulting in the
construction of SynSciPass. By training the same model that performed well on
DAGPap22 on SynSciPass, we show that not only is the model more robust to
domain shifts but also is able to uncover the type of technology used for
machine generated text. Despite this, we conclude that current datasets are
neither comprehensive nor realistic enough to understand how these models would
perform in the wild where manuscript submissions can come from many unknown or
novel distributions, how they would perform on scientific full-texts rather
than small passages, and what might happen when there is a mix of appropriate
and inappropriate uses of natural language generation.
