  Generative AI including large language models (LLMs) has recently gained
significant interest in the geo-science community through its versatile
task-solving capabilities including programming, arithmetic reasoning,
generation of sample data, time-series forecasting, toponym recognition, or
image classification. Most existing performance assessments of LLMs for spatial
tasks have primarily focused on ChatGPT, whereas other chatbots received less
attention. To narrow this research gap, this study conducts a zero-shot
correctness evaluation for a set of 76 spatial tasks across seven task
categories assigned to four prominent chatbots, i.e., ChatGPT-4, Gemini,
Claude-3, and Copilot. The chatbots generally performed well on tasks related
to spatial literacy, GIS theory, and interpretation of programming code and
functions, but revealed weaknesses in mapping, code writing, and spatial
reasoning. Furthermore, there was a significant difference in correctness of
results between the four chatbots. Responses from repeated tasks assigned to
each chatbot showed a high level of consistency in responses with matching
rates of over 80% for most task categories in the four chatbots.
