  Visual grounding of Language aims at enriching textual representations of
language with multiple sources of visual knowledge such as images and videos.
Although visual grounding is an area of intense research, inter-lingual aspects
of visual grounding have not received much attention. The present study
investigates the inter-lingual visual grounding of word embeddings. We propose
an implicit alignment technique between the two spaces of vision and language
in which inter-lingual textual information interacts in order to enrich
pre-trained textual word embeddings. We focus on three languages in our
experiments, namely, English, Arabic, and German. We obtained visually grounded
vector representations for these languages and studied whether visual grounding
on one or multiple languages improved the performance of embeddings on word
similarity and categorization benchmarks. Our experiments suggest that
inter-lingual knowledge improves the performance of grounded embeddings in
similar languages such as German and English. However, inter-lingual grounding
of German or English with Arabic led to a slight degradation in performance on
word similarity benchmarks. On the other hand, we observed an opposite trend on
categorization benchmarks where Arabic had the most improvement on English. In
the discussion section, several reasons for those findings are laid out. We
hope that our experiments provide a baseline for further research on
inter-lingual visual grounding.
