  High Efficiency Video Coding (HEVC) significantly reduces bit-rates over the
proceeding H.264 standard but at the expense of extremely high encoding
complexity. In HEVC, the quad-tree partition of coding unit (CU) consumes a
large proportion of the HEVC encoding complexity, due to the bruteforce search
for rate-distortion optimization (RDO). Therefore, this paper proposes a deep
learning approach to predict the CU partition for reducing the HEVC complexity
at both intra- and inter-modes, which is based on convolutional neural network
(CNN) and long- and short-term memory (LSTM) network. First, we establish a
large-scale database including substantial CU partition data for HEVC intra-
and inter-modes. This enables deep learning on the CU partition. Second, we
represent the CU partition of an entire coding tree unit (CTU) in the form of a
hierarchical CU partition map (HCPM). Then, we propose an early-terminated
hierarchical CNN (ETH-CNN) for learning to predict the HCPM. Consequently, the
encoding complexity of intra-mode HEVC can be drastically reduced by replacing
the brute-force search with ETH-CNN to decide the CU partition. Third, an
early-terminated hierarchical LSTM (ETH-LSTM) is proposed to learn the temporal
correlation of the CU partition. Then, we combine ETH-LSTM and ETH-CNN to
predict the CU partition for reducing the HEVC complexity for inter-mode.
Finally, experimental results show that our approach outperforms other
state-of-the-art approaches in reducing the HEVC complexity at both intra- and
inter-modes.
