  With deep learning (DL) outperforming conventional methods for different
tasks, much effort has been devoted to utilizing DL in various domains.
Researchers and developers in the traffic domain have also designed and
improved DL models for forecasting tasks such as estimation of traffic speed
and time of arrival. However, there exist many challenges in analyzing DL
models due to the black-box property of DL models and complexity of traffic
data (i.e., spatio-temporal dependencies). Collaborating with domain experts,
we design a visual analytics system, AttnAnalyzer, that enables users to
explore how DL models make predictions by allowing effective spatio-temporal
dependency analysis. The system incorporates dynamic time warping (DTW) and
Granger causality tests for computational spatio-temporal dependency analysis
while providing map, table, line chart, and pixel views to assist user to
perform dependency and model behavior analysis. For the evaluation, we present
three case studies showing how AttnAnalyzer can effectively explore model
behaviors and improve model performance in two different road networks. We also
provide domain expert feedback.
