  In the first half of 2023, text-generative artificial intelligence (AI),
including ChatGPT, equipped with GPT-3.5 and GPT-4, from OpenAI, has attracted
considerable attention worldwide. In this study, first, we compared Japanese
stylometric features of texts generated by GPT (-3.5 and -4) and those written
by humans. In this work, we performed multi-dimensional scaling (MDS) to
confirm the distributions of 216 texts of three classes (72 academic papers
written by 36 single authors, 72 texts generated by GPT-3.5, and 72 texts
generated by GPT-4 on the basis of the titles of the aforementioned papers)
focusing on the following stylometric features: (1) bigrams of parts-of-speech,
(2) bigram of postpositional particle words, (3) positioning of commas, and (4)
rate of function words. MDS revealed distinct distributions at each stylometric
feature of GPT (-3.5 and -4) and human. Although GPT-4 is more powerful than
GPT-3.5 because it has more parameters, both GPT (-3.5 and -4) distributions
are likely to overlap. These results indicate that although the number of
parameters may increase in the future, GPT-generated texts may not be close to
that written by humans in terms of stylometric features. Second, we verified
the classification performance of random forest (RF) for two classes (GPT and
human) focusing on Japanese stylometric features. This study revealed the high
performance of RF in each stylometric feature: The RF classifier focusing on
the rate of function words achieved 98.1% accuracy. Furthermore the RF
classifier focusing on all stylometric features reached 100% in terms of all
performance indexes (accuracy, recall, precision, and F1 score). This study
concluded that at this stage we human discriminate ChatGPT from human limited
to Japanese language.
