  Video Quality Assessment (VQA), which aims to predict the perceptual quality
of a video, has attracted raising attention with the rapid development of
streaming media technology, such as Facebook, TikTok, Kwai, and so on. Compared
with other sequence-based visual tasks (\textit{e.g.,} action recognition), VQA
faces two under-estimated challenges unresolved in User Generated Content (UGC)
videos. \textit{First}, it is not rare that several frames containing serious
distortions (\textit{e.g.,}blocking, blurriness), can determine the perceptual
quality of the whole video, while other sequence-based tasks require more
frames of equal importance for representations. \textit{Second}, the perceptual
quality of a video exhibits a multi-distortion distribution, due to the
differences in the duration and probability of occurrence for various
distortions. In order to solve the above challenges, we propose \textit{Visual
Quality Transformer (VQT)} to extract quality-related sparse features more
efficiently. Methodologically, a Sparse Temporal Attention (STA) is proposed to
sample keyframes by analyzing the temporal correlation between frames, which
reduces the computational complexity from $O(T^2)$ to $O(T \log T)$.
Structurally, a Multi-Pathway Temporal Network (MPTN) utilizes multiple STA
modules with different degrees of sparsity in parallel, capturing co-existing
distortions in a video. Experimentally, VQT demonstrates superior performance
than many \textit{state-of-the-art} methods in three public no-reference VQA
datasets. Furthermore, VQT shows better performance in four full-reference VQA
datasets against widely-adopted industrial algorithms (\textit{i.e.,} VMAF and
AVQT).
