  Most existing action quality assessment methods rely on the deep features of
an entire video to predict the score, which is less reliable due to the
non-transparent inference process and poor interpretability. We argue that
understanding both high-level semantics and internal temporal structures of
actions in competitive sports videos is the key to making predictions accurate
and interpretable. Towards this goal, we construct a new fine-grained dataset,
called FineDiving, developed on diverse diving events with detailed annotations
on action procedures. We also propose a procedure-aware approach for action
quality assessment, learned by a new Temporal Segmentation Attention module.
Specifically, we propose to parse pairwise query and exemplar action instances
into consecutive steps with diverse semantic and temporal correspondences. The
procedure-aware cross-attention is proposed to learn embeddings between query
and exemplar steps to discover their semantic, spatial, and temporal
correspondences, and further serve for fine-grained contrastive regression to
derive a reliable scoring mechanism. Extensive experiments demonstrate that our
approach achieves substantial improvements over state-of-the-art methods with
better interpretability. The dataset and code are available at
\url{https://github.com/xujinglin/FineDiving}.
