  Multimodal large language models (MLLMs) have shown remarkable capabilities
across a broad range of tasks but their knowledge and abilities in the
geographic and geospatial domains are yet to be explored, despite potential
wide-ranging benefits to navigation, environmental research, urban development,
and disaster response. We conduct a series of experiments exploring various
vision capabilities of MLLMs within these domains, particularly focusing on the
frontier model GPT-4V, and benchmark its performance against open-source
counterparts. Our methodology involves challenging these models with a
small-scale geographic benchmark consisting of a suite of visual tasks, testing
their abilities across a spectrum of complexity. The analysis uncovers not only
where such models excel, including instances where they outperform humans, but
also where they falter, providing a balanced view of their capabilities in the
geographic domain. To enable the comparison and evaluation of future models,
our benchmark will be publicly released.
