  We propose a multi-layer variational autoencoder method, we call HR-VQVAE,
that learns hierarchical discrete representations of the data. By utilizing a
novel objective function, each layer in HR-VQVAE learns a discrete
representation of the residual from previous layers through a vector quantized
encoder. Furthermore, the representations at each layer are hierarchically
linked to those at previous layers. We evaluate our method on the tasks of
image reconstruction and generation. Experimental results demonstrate that the
discrete representations learned by HR-VQVAE enable the decoder to reconstruct
high-quality images with less distortion than the baseline methods, namely
VQVAE and VQVAE-2. HR-VQVAE can also generate high-quality and diverse images
that outperform state-of-the-art generative models, providing further
verification of the efficiency of the learned representations. The hierarchical
nature of HR-VQVAE i) reduces the decoding search time, making the method
particularly suitable for high-load tasks and ii) allows to increase the
codebook size without incurring the codebook collapse problem.
