  The idea that social media platforms like Twitter are inhabited by vast
numbers of social bots has become widely accepted in recent years. Social bots
are assumed to be automated social media accounts operated by malicious actors
with the goal of manipulating public opinion. They are credited with the
ability to produce content autonomously and to interact with human users.
Social bot activity has been reported in many different political contexts,
including the U.S. presidential elections, discussions about migration, climate
change, and COVID-19. However, the relevant publications either use crude and
questionable heuristics to discriminate between supposed social bots and humans
or -- in the vast majority of the cases -- fully rely on the output of
automatic bot detection tools, most commonly Botometer. In this paper, we point
out a fundamental theoretical flaw in the widely-used study design for
estimating the prevalence of social bots. Furthermore, we empirically
investigate the validity of peer-reviewed Botometer-based studies by closely
and systematically inspecting hundreds of accounts that had been counted as
social bots. We were unable to find a single social bot. Instead, we found
mostly accounts undoubtedly operated by human users, the vast majority of them
using Twitter in an inconspicuous and unremarkable fashion without the
slightest traces of automation. We conclude that studies claiming to
investigate the prevalence, properties, or influence of social bots based on
Botometer have, in reality, just investigated false positives and artifacts of
this approach.
