  In recent years, machine learning (ML) has relied heavily on crowdworkers
both for building datasets and for addressing research questions requiring
human interaction or judgment. The diverse tasks performed and uses of the data
produced render it difficult to determine when crowdworkers are best thought of
as workers (versus human subjects). These difficulties are compounded by
conflicting policies, with some institutions and researchers regarding all ML
crowdworkers as human subjects and others holding that they rarely constitute
human subjects. Notably few ML papers involving crowdwork mention IRB
oversight, raising the prospect of non-compliance with ethical and regulatory
requirements. We investigate the appropriate designation of ML crowdsourcing
studies, focusing our inquiry on natural language processing to expose unique
challenges for research oversight. Crucially, under the U.S. Common Rule, these
judgments hinge on determinations of aboutness, concerning both whom (or what)
the collected data is about and whom (or what) the analysis is about. We
highlight two challenges posed by ML: the same set of workers can serve
multiple roles and provide many sorts of information; and ML research tends to
embrace a dynamic workflow, where research questions are seldom stated ex ante
and data sharing opens the door for future studies to aim questions at
different targets. Our analysis exposes a potential loophole in the Common
Rule, where researchers can elude research ethics oversight by splitting data
collection and analysis into distinct studies. Finally, we offer several policy
recommendations to address these concerns.
