  Many objects in the real world undergo dramatic variations in visual
appearance. For example, a tomato may be red or green, sliced or chopped, fresh
or fried, liquid or solid. Training a single detector to accurately recognize
tomatoes in all these different states is challenging. On the other hand,
contextual cues (e.g., the presence of a knife, a cutting board, a strainer or
a pan) are often strongly indicative of how the object appears in the scene.
Recognizing such contextual cues is useful not only to improve the accuracy of
object detection or to determine the state of the object, but also to
understand its functional properties and to infer ongoing or upcoming
human-object interactions. A fully-supervised approach to recognizing object
states and their contexts in the real-world is unfortunately marred by the
long-tailed, open-ended distribution of the data, which would effectively
require massive amounts of annotations to capture the appearance of objects in
all their different forms. Instead of relying on manually-labeled data for this
task, we propose a new framework for learning Contextualized OBject Embeddings
(COBE) from automatically-transcribed narrations of instructional videos. We
leverage the semantic and compositional structure of language by training a
visual detector to predict a contextualized word embedding of the object and
its associated narration. This enables the learning of an object representation
where concepts relate according to a semantic language metric. Our experiments
show that our detector learns to predict a rich variety of contextual object
information, and that it is highly effective in the settings of few-shot and
zero-shot learning.
