  Real-time detection and prediction of extreme weather protect human lives and
infrastructure. Traditional methods rely on numerical threshold setting and
manual interpretation of weather heatmaps with Geographic Information Systems
(GIS), which can be slow and error-prone. Our research redefines Extreme
Weather Events Detection (EWED) by framing it as a Visual Question Answering
(VQA) problem, thereby introducing a more precise and automated solution.
Leveraging Vision-Language Models (VLM) to simultaneously process visual and
textual data, we offer an effective aid to enhance the analysis process of
weather heatmaps. Our initial assessment of general-purpose VLMs (e.g.,
GPT-4-Vision) on EWED revealed poor performance, characterized by low accuracy
and frequent hallucinations due to inadequate color differentiation and
insufficient meteorological knowledge. To address these challenges, we
introduce ClimateIQA, the first meteorological VQA dataset, which includes
8,760 wind gust heatmaps and 254,040 question-answer pairs covering four
question types, both generated from the latest climate reanalysis data. We also
propose Sparse Position and Outline Tracking (SPOT), an innovative technique
that leverages OpenCV and K-Means clustering to capture and depict color
contours in heatmaps, providing ClimateIQA with more accurate color spatial
location information. Finally, we present Climate-Zoo, the first meteorological
VLM collection, which adapts VLMs to meteorological applications using the
ClimateIQA dataset. Experiment results demonstrate that models from Climate-Zoo
substantially outperform state-of-the-art general VLMs, achieving an accuracy
increase from 0% to over 90% in EWED verification. The datasets and models in
this study are publicly available for future climate science research:
https://github.com/AlexJJJChen/Climate-Zoo.
