  Visually-grounded spoken language datasets can enable models to learn
cross-modal correspondences with very weak supervision. However, modern
audio-visual datasets contain biases that undermine the real-world performance
of models trained on that data. We introduce Spoken ObjectNet, which is
designed to remove some of these biases and provide a way to better evaluate
how effectively models will perform in real-world scenarios. This dataset
expands upon ObjectNet, which is a bias-controlled image dataset that features
similar image classes to those present in ImageNet. We detail our data
collection pipeline, which features several methods to improve caption quality,
including automated language model checks. Lastly, we show baseline results on
image retrieval and audio retrieval tasks. These results show that models
trained on other datasets and then evaluated on Spoken ObjectNet tend to
perform poorly due to biases in other datasets that the models have learned. We
also show evidence that the performance decrease is due to the dataset
controls, and not the transfer setting.
