  Recent computational models of the acquisition of spoken language via
grounding in perception exploit associations between the spoken and visual
modalities and learn to represent speech and visual data in a joint vector
space. A major unresolved issue from the point of ecological validity is the
training data, typically consisting of images or videos paired with spoken
descriptions of what is depicted. Such a setup guarantees an unrealistically
strong correlation between speech and the visual data. In the real world the
coupling between the linguistic and the visual modality is loose, and often
confounded by correlations with non-semantic aspects of the speech signal. Here
we address this shortcoming by using a dataset based on the children's cartoon
Peppa Pig. We train a simple bi-modal architecture on the portion of the data
consisting of dialog between characters, and evaluate on segments containing
descriptive narrations. Despite the weak and confounded signal in this training
data our model succeeds at learning aspects of the visual semantics of spoken
language.
