  Most existing traffic sign-related works are dedicated to detecting and
recognizing part of traffic signs individually, which fails to analyze the
global semantic logic among signs and may convey inaccurate traffic
instruction. Following the above issues, we propose a traffic sign
interpretation (TSI) task, which aims to interpret global semantic interrelated
traffic signs (e.g.,~driving instruction-related texts, symbols, and guide
panels) into a natural language for providing accurate instruction support to
autonomous or assistant driving. Meanwhile, we design a multi-task learning
architecture for TSI, which is responsible for detecting and recognizing
various traffic signs and interpreting them into a natural language like a
human. Furthermore, the absence of a public TSI available dataset prompts us to
build a traffic sign interpretation dataset, namely TSI-CN. The dataset
consists of real road scene images, which are captured from the highway and the
urban way in China from a driver's perspective. It contains rich location
labels of texts, symbols, and guide panels, and the corresponding natural
language description labels. Experiments on TSI-CN demonstrate that the TSI
task is achievable and the TSI architecture can interpret traffic signs from
scenes successfully even if there is a complex semantic logic among signs. The
TSI-CN dataset and the source code of the TSI architecture will be publicly
available after the revision process.
