  We present "Human or Not?", an online game inspired by the Turing test, that
measures the capability of AI chatbots to mimic humans in dialog, and of humans
to tell bots from other humans. Over the course of a month, the game was played
by over 1.5 million users who engaged in anonymous two-minute chat sessions
with either another human or an AI language model which was prompted to behave
like humans. The task of the players was to correctly guess whether they spoke
to a person or to an AI. This largest scale Turing-style test conducted to date
revealed some interesting facts. For example, overall users guessed the
identity of their partners correctly in only 68% of the games. In the subset of
the games in which users faced an AI bot, users had even lower correct guess
rates of 60% (that is, not much higher than chance). This white paper details
the development, deployment, and results of this unique experiment. While this
experiment calls for many extensions and refinements, these findings already
begin to shed light on the inevitable near future which will commingle humans
and AI.
