  As an important and challenging problem in computer vision, PAnoramic
Semantic Segmentation (PASS) gives complete scene perception based on an
ultra-wide angle of view. Usually, prevalent PASS methods with 2D panoramic
image input focus on solving image distortions but lack consideration of the 3D
properties of original $360^{\circ}$ data. Therefore, their performance will
drop a lot when inputting panoramic images with the 3D disturbance. To be more
robust to 3D disturbance, we propose our Spherical Geometry-Aware Transformer
for PAnoramic Semantic Segmentation (SGAT4PASS), considering 3D spherical
geometry knowledge. Specifically, a spherical geometry-aware framework is
proposed for PASS. It includes three modules, i.e., spherical geometry-aware
image projection, spherical deformable patch embedding, and a panorama-aware
loss, which takes input images with 3D disturbance into account, adds a
spherical geometry-aware constraint on the existing deformable patch embedding,
and indicates the pixel density of original $360^{\circ}$ data, respectively.
Experimental results on Stanford2D3D Panoramic datasets show that SGAT4PASS
significantly improves performance and robustness, with approximately a 2%
increase in mIoU, and when small 3D disturbances occur in the data, the
stability of our performance is improved by an order of magnitude. Our code and
supplementary material are available at
https://github.com/TencentARC/SGAT4PASS.
