  Machine learning (ML) enabled classification models are becoming increasingly
popular for tackling the sheer volume and speed of online misinformation and
other content that could be identified as harmful. In building these models,
data scientists need to take a stance on the legitimacy, authoritativeness and
objectivity of the sources of ``truth" used for model training and testing.
This has political, ethical and epistemic implications which are rarely
addressed in technical papers. Despite (and due to) their reported high
accuracy and performance, ML-driven moderation systems have the potential to
shape online public debate and create downstream negative impacts such as undue
censorship and the reinforcing of false beliefs. Using collaborative
ethnography and theoretical insights from social studies of science and
expertise, we offer a critical analysis of the process of building ML models
for (mis)information classification: we identify a series of algorithmic
contingencies--key moments during model development that could lead to
different future outcomes, uncertainty and harmful effects as these tools are
deployed by social media platforms. We conclude by offering a tentative path
toward reflexive and responsible development of ML tools for moderating
misinformation and other harmful content online.
