  Self-supervised learning has emerged as a powerful tool for depth and
ego-motion estimation, leading to state-of-the-art results on benchmark
datasets. However, one significant limitation shared by current methods is the
assumption of a known parametric camera model -- usually the standard pinhole
geometry -- leading to failure when applied to imaging systems that deviate
significantly from this assumption (e.g., catadioptric cameras or underwater
imaging). In this work, we show that self-supervision can be used to learn
accurate depth and ego-motion estimation without prior knowledge of the camera
model. Inspired by the geometric model of Grossberg and Nayar, we introduce
Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise
projection rays, approximating a wide range of cameras. NRS are fully
differentiable and can be learned end-to-end from unlabeled raw videos. We
demonstrate the use of NRS for self-supervised learning of visual odometry and
depth estimation from raw videos obtained using a wide variety of camera
systems, including pinhole, fisheye, and catadioptric.
