  Sparse meteorological forecasting is indispensable for fine-grained weather
forecasting and deserves extensive attention. Recent studies have highlighted
the potential of spatio-temporal graph convolutional networks (ST-GCNs) in
predicting numerical data from ground weather stations. However, as one of the
highest fidelity and lowest latency data, the application of the vision data
from satellites in ST-GCNs remains unexplored. There are few studies to
demonstrate the effectiveness of combining the above multi-modal data for
sparse meteorological forecasting. Towards this objective, we introduce
Vision-Numerical Fusion Graph Convolutional Network (VN-Net), which mainly
utilizes: 1) Numerical-GCN (N-GCN) to adaptively model the static and dynamic
patterns of spatio-temporal numerical data; 2) Vision-LSTM Network (V-LSTM) to
capture multi-scale joint channel and spatial features from time series
satellite images; 4) a GCN-based decoder to generate hourly predictions of
specified meteorological factors. As far as we know, VN-Net is the first
attempt to introduce GCN method to utilize multi-modal data for better handling
sparse spatio-temporal meteorological forecasting. Our experiments on Weather2k
dataset show VN-Net outperforms state-of-the-art by a significant margin on
mean absolute error (MAE) and root mean square error (RMSE) for temperature,
relative humidity, and visibility forecasting. Furthermore, we conduct
interpretation analysis and design quantitative evaluation metrics to assess
the impact of incorporating vision data.
