  Recently, transformer-based methods have gained significant success in
sequential 2D-to-3D lifting human pose estimation. As a pioneering work,
PoseFormer captures spatial relations of human joints in each video frame and
human dynamics across frames with cascaded transformer layers and has achieved
impressive performance. However, in real scenarios, the performance of
PoseFormer and its follow-ups is limited by two factors: (a) The length of the
input joint sequence; (b) The quality of 2D joint detection. Existing methods
typically apply self-attention to all frames of the input sequence, causing a
huge computational burden when the frame number is increased to obtain advanced
estimation accuracy, and they are not robust to noise naturally brought by the
limited capability of 2D joint detectors. In this paper, we propose
PoseFormerV2, which exploits a compact representation of lengthy skeleton
sequences in the frequency domain to efficiently scale up the receptive field
and boost robustness to noisy 2D joint detection. With minimum modifications to
PoseFormer, the proposed method effectively fuses features both in the time
domain and frequency domain, enjoying a better speed-accuracy trade-off than
its precursor. Extensive experiments on two benchmark datasets (i.e., Human3.6M
and MPI-INF-3DHP) demonstrate that the proposed approach significantly
outperforms the original PoseFormer and other transformer-based variants. Code
is released at \url{https://github.com/QitaoZhao/PoseFormerV2}.
