  Vision-Language Models (VLMs) are expected to be capable of reasoning with
commonsense knowledge as human beings. One example is that humans can reason
where and when an image is taken based on their knowledge. This makes us wonder
if, based on visual cues, Vision-Language Models that are pre-trained with
large-scale image-text resources can achieve and even outperform human's
capability in reasoning times and location. To address this question, we
propose a two-stage \recognition\space and \reasoning\space probing task,
applied to discriminative and generative VLMs to uncover whether VLMs can
recognize times and location-relevant features and further reason about it. To
facilitate the investigation, we introduce WikiTiLo, a well-curated image
dataset compromising images with rich socio-cultural cues. In the extensive
experimental studies, we find that although VLMs can effectively retain
relevant features in visual encoders, they still fail to make perfect
reasoning. We will release our dataset and codes to facilitate future studies.
