  A Manhattan world lying along cuboid buildings is useful for camera angle
estimation. However, accurate and robust angle estimation from fisheye images
in the Manhattan world has remained an open challenge because general scene
images tend to lack constraints such as lines, arcs, and vanishing points. To
achieve higher accuracy and robustness, we propose a learning-based calibration
method that uses heatmap regression, which is similar to pose estimation using
keypoints, to detect the directions of labeled image coordinates.
Simultaneously, our two estimators recover the rotation and remove fisheye
distortion by remapping from a general scene image. Without considering
vanishing-point constraints, we find that additional points for learning-based
methods can be defined. To compensate for the lack of vanishing points in
images, we introduce auxiliary diagonal points that have the optimal 3D
arrangement of spatial uniformity. Extensive experiments demonstrated that our
method outperforms conventional methods on large-scale datasets and with
off-the-shelf cameras.
