  Conditional coding has lately emerged as the mainstream approach to learned
video compression. However, a recent study shows that it may perform worse than
residual coding when the information bottleneck arises. Conditional residual
coding was thus proposed, creating a new school of thought to improve on
conditional coding. Notably, conditional residual coding relies heavily on the
assumption that the residual frame has a lower entropy rate than that of the
intra frame. Recognizing that this assumption is not always true due to
dis-occlusion phenomena or unreliable motion estimates, we propose a masked
conditional residual coding scheme. It learns a soft mask to form a hybrid of
conditional coding and conditional residual coding in a pixel adaptive manner.
We introduce a Transformer-based conditional autoencoder. Several strategies
are investigated with regard to how to condition a Transformer-based
autoencoder for inter-frame coding, a topic that is largely under-explored.
Additionally, we propose a channel transform module (CTM) to decorrelate the
image latents along the channel dimension, with the aim of using the simple
hyperprior to approach similar compression performance to the channel-wise
autoregressive model. Experimental results confirm the superiority of our
masked conditional residual transformer (termed MaskCRT) to both conditional
coding and conditional residual coding. On commonly used datasets, MaskCRT
shows comparable BD-rate results to VTM-17.0 under the low delay P
configuration in terms of PSNR-RGB and outperforms VTM-17.0 in terms of
MS-SSIM-RGB. It also opens up a new research direction for advancing learned
video compression.
