  In this paper, we aim to address issues of (1) joint spatial-temporal
modeling and (2) side information injection for deep-learning based in-loop
filter. For (1), we design a deep network with both progressive rethinking and
collaborative learning mechanisms to improve quality of the reconstructed
intra-frames and inter-frames, respectively. For intra coding, a Progressive
Rethinking Network (PRN) is designed to simulate the human decision mechanism
for effective spatial modeling. Our designed block introduces an additional
inter-block connection to bypass a high-dimensional informative feature before
the bottleneck module across blocks to review the complete past memorized
experiences and rethinks progressively. For inter coding, the current
reconstructed frame interacts with reference frames (peak quality frame and the
nearest adjacent frame) collaboratively at the feature level. For (2), we
extract both intra-frame and inter-frame side information for better context
modeling. A coarse-to-fine partition map based on HEVC partition trees is built
as the intra-frame side information. Furthermore, the warped features of the
reference frames are offered as the inter-frame side information. Our PRN with
intra-frame side information provides 9.0% BD-rate reduction on average
compared to HEVC baseline under All-intra (AI) configuration. While under
Low-Delay B (LDB), Low-Delay P (LDP) and Random Access (RA) configuration, our
PRN with inter-frame side information provides 9.0%, 10.6% and 8.0% BD-rate
reduction on average respectively. Our project webpage is
https://dezhao-wang.github.io/PRN-v2/.
