  Interactions between humans are diverse and context-dependent, but previous
works have treated them as categorical, disregarding the heavy tail of possible
interactions. We propose a new paradigm of learning human-human interactions as
free text from a single still image, allowing for flexibility in modeling the
unlimited space of situations and relationships between people. To overcome the
absence of data labelled specifically for this task, we use knowledge
distillation applied to synthetic caption data produced by a large language
model without explicit supervision. We show that the pseudo-labels produced by
this procedure can be used to train a captioning model to effectively
understand human-human interactions in images, as measured by a variety of
metrics that measure textual and semantic faithfulness and factual groundedness
of our predictions. We further show that our approach outperforms SOTA image
captioning and situation recognition models on this task. We will release our
code and pseudo-labels along with Waldo and Wenda, a manually-curated test set
for still image human-human interaction understanding.
