  In an often-cited 2019 paper on the use of machine learning in political
research, Anastasopoulos & Whitford (A&W) propose a text classification method
for tweets related to organizational reputation. The aim of their paper was to
provide a 'guide to practice' for public administration scholars and
practitioners on the use of machine learning. In the current paper we follow up
on that work with a replication of A&W's experiments and additional analyses on
model stability and the effects of preprocessing, both in relation to the small
data size. We show that (1) the small data causes the classification model to
be highly sensitive to variations in the random train-test split, and that (2)
the applied preprocessing causes the data to be extremely sparse, with the
majority of items in the data having at most two non-zero lexical features.
With additional experiments in which we vary the steps of the preprocessing
pipeline, we show that the small data size keeps causing problems, irrespective
of the preprocessing choices. Based on our findings, we argue that A&W's
conclusions regarding the automated classification of organizational reputation
tweets -- either substantive or methodological -- can not be maintained and
require a larger data set for training and more careful validation.
