  It has been found that temporal action proposal generation, which aims to
discover the temporal action instances within the range of the start and end
frames in the untrimmed videos, can largely benefit from proper temporal and
semantic context exploitation. The latest efforts were dedicated to considering
the temporal context and similarity-based semantic contexts through
self-attention modules. However, they still suffer from cluttered background
information and limited contextual feature learning. In this paper, we propose
a novel Pyramid Region-based Slot Attention (PRSlot) module to address these
issues. Instead of using the similarity computation, our PRSlot module directly
learns the local relations in an encoder-decoder manner and generates the
representation of a local region enhanced based on the attention over input
features called \textit{slot}. Specifically, upon the input snippet-level
features, PRSlot module takes the target snippet as \textit{query}, its
surrounding region as \textit{key} and then generates slot representations for
each \textit{query-key} slot by aggregating the local snippet context with a
parallel pyramid strategy. Based on PRSlot modules, we present a novel Pyramid
Region-based Slot Attention Network termed PRSA-Net to learn a unified visual
representation with rich temporal and semantic context for better proposal
generation. Extensive experiments are conducted on two widely adopted THUMOS14
and ActivityNet-1.3 benchmarks. Our PRSA-Net outperforms other state-of-the-art
methods. In particular, we improve the AR@100 from the previous best 50.67% to
56.12% for proposal generation and raise the mAP under 0.5 tIoU from 51.9\% to
58.7\% for action detection on THUMOS14. \textit{Code is available at}
\url{https://github.com/handhand123/PRSA-Net}
