  Recent work considered how images paired with speech can be used as
supervision for building speech systems when transcriptions are not available.
We ask whether visual grounding can be used for cross-lingual keyword spotting:
given a text keyword in one language, the task is to retrieve spoken utterances
containing that keyword in another language. This could enable searching
through speech in a low-resource language using text queries in a high-resource
language. As a proof-of-concept, we use English speech with German queries: we
use a German visual tagger to add keyword labels to each training image, and
then train a neural network to map English speech to German keywords. Without
seeing parallel speech-transcriptions or translations, the model achieves a
precision at ten of 58%. We show that most erroneous retrievals contain
equivalent or semantically relevant keywords; excluding these would improve
P@10 to 91%.
