  We present our work in progress exploring the possibilities of a shared
embedding space between textual and visual modality. Leveraging the textual
nature of object detection labels and the hypothetical expressiveness of
extracted visual object representations, we propose an approach opposite to the
current trend, grounding of the representations in the word embedding space of
the captioning system instead of grounding words or sentences in their
associated images. Based on the previous work, we apply additional grounding
losses to the image captioning training objective aiming to force visual object
representations to create more heterogeneous clusters based on their class
label and copy a semantic structure of the word embedding space. In addition,
we provide an analysis of the learned object vector space projection and its
impact on the IC system performance. With only slight change in performance,
grounded models reach the stopping criterion during training faster than the
unconstrained model, needing about two to three times less training updates.
Additionally, an improvement in structural correlation between the word
embeddings and both original and projected object vectors suggests that the
grounding is actually mutual.
