  Learning a common representation space between vision and language allows
deep networks to relate objects in the image to the corresponding semantic
meaning. We present a model that learns a shared Gaussian mixture
representation imposing the compositionality of the text onto the visual domain
without having explicit location supervision. By combining the spatial
transformer with a representation learning approach we learn to split images
into separately encoded patches to associate visual and textual representations
in an interpretable manner. On variations of MNIST and CIFAR10, our model is
able to perform weakly supervised object detection and demonstrates its ability
to extrapolate to unseen combination of objects.
