  Visually grounded speech (VGS) models are trained on images paired with
unlabelled spoken captions. Such models could be used to build speech systems
in settings where it is impossible to get labelled data, e.g. for documenting
unwritten languages. However, most VGS studies are in English or other
high-resource languages. This paper attempts to address this shortcoming. We
collect and release a new single-speaker dataset of audio captions for 6k
Flickr images in Yor\`ub\'a -- a real low-resource language spoken in Nigeria.
We train an attention-based VGS model where images are automatically tagged
with English visual labels and paired with Yor\`ub\'a utterances. This enables
cross-lingual keyword localisation: a written English query is detected and
located in Yor\`ub\'a speech. To quantify the effect of the smaller dataset, we
compare to English systems trained on similar and more data. We hope that this
new dataset will stimulate research in the use of VGS models for real
low-resource languages.
