  Interactive machine learning (IML) allows users to build their custom machine
learning models without expert knowledge. While most existing IML systems are
designed with classification algorithms, they sometimes oversimplify the
capabilities of machine learning algorithms and restrict the user's task
definition. On the other hand, as recent large-scale language models have
shown, natural language representation has the potential to enable more
flexible and generic task descriptions. Models that take images as input and
output text have the potential to represent a variety of tasks by providing
appropriate text labels for training. However, the effect of introducing text
labels to IML system design has never been investigated. In this work, we aim
to investigate the difference between image-to-text translation and image
classification for IML systems. Using our prototype systems, we conducted a
comparative user study with non-expert users, where participants solved various
tasks. Our results demonstrate the underlying difficulty for users in properly
defining image recognition tasks while highlighting the potential and
challenges of interactive image-to-text translation systems.
