  This paper presents a novel line-aware rectification network (LaRecNet) to
address the problem of fisheye distortion rectification based on the classical
observation that straight lines in 3D space should be still straight in image
planes. Specifically, the proposed LaRecNet contains three sequential modules
to (1) learn the distorted straight lines from fisheye images; (2) estimate the
distortion parameters from the learned heatmaps and the image appearance; and
(3) rectify the input images via a proposed differentiable rectification layer.
To better train and evaluate the proposed model, we create a synthetic
line-rich fisheye (SLF) dataset that contains the distortion parameters and
well-annotated distorted straight lines of fisheye images. The proposed method
enables us to simultaneously calibrate the geometric distortion parameters and
rectify fisheye images. Extensive experiments demonstrate that our model
achieves state-of-the-art performance in terms of both geometric accuracy and
image quality on several evaluation metrics. In particular, the images
rectified by LaRecNet achieve an average reprojection error of 0.33 pixels on
the SLF dataset and produce the highest peak signal-to-noise ratio (PSNR) and
structure similarity index (SSIM) compared with the groundtruth.
