  The application of computer vision to nuanced subjective use cases is
growing. While crowdsourcing has served the vision community well for most
objective tasks (such as labeling a "zebra"), it now falters on tasks where
there is substantial subjectivity in the concept (such as identifying "gourmet
tuna"). However, empowering any user to develop a classifier for their concept
is technically difficult: users are neither machine learning experts, nor have
the patience to label thousands of examples. In reaction, we introduce the
problem of Agile Modeling: the process of turning any subjective visual concept
into a computer vision model through a real-time user-in-the-loop interactions.
We instantiate an Agile Modeling prototype for image classification and show
through a user study (N=14) that users can create classifiers with minimal
effort under 30 minutes. We compare this user driven process with the
traditional crowdsourcing paradigm and find that the crowd's notion often
differs from that of the user's, especially as the concepts become more
subjective. Finally, we scale our experiments with simulations of users
training classifiers for ImageNet21k categories to further demonstrate the
efficacy.
