  Recent neural language models have taken a significant step forward in
producing remarkably controllable, fluent, and grammatical text. Although
studies have found that AI-generated text is not distinguishable from
human-written text for crowd-sourcing workers, there still exist errors in
AI-generated text which are even subtler and harder to spot. We primarily focus
on the scenario in which scientific AI writing assistant is deeply involved.
First, we construct a feature description framework to distinguish between
AI-generated text and human-written text from syntax, semantics, and pragmatics
based on the human evaluation. Then we utilize the features, i.e., writing
style, coherence, consistency, and argument logistics, from the proposed
framework to analyze two types of content. Finally, we adopt several publicly
available methods to investigate the gap of between AI-generated scientific
text and human-written scientific text by AI-generated scientific text
detection models. The results suggest that while AI has the potential to
generate scientific content that is as accurate as human-written content, there
is still a gap in terms of depth and overall quality. The AI-generated
scientific content is more likely to contain errors in factual issues. We find
that there exists a "writing style" gap between AI-generated scientific text
and human-written scientific text. Based on the analysis result, we summarize a
series of model-agnostic and distribution-agnostic features for detection tasks
in other domains. Findings in this paper contribute to guiding the optimization
of AI models to produce high-quality content and addressing related ethical and
security concerns.
