  Visual-inertial-odometry has attracted extensive attention in the field of
autonomous driving and robotics. The size of Field of View (FoV) plays an
important role in Visual-Odometry (VO) and Visual-Inertial-Odometry (VIO), as a
large FoV enables to perceive a wide range of surrounding scene elements and
features. However, when the field of the camera reaches the negative half
plane, one cannot simply use [u,v,1]^T to represent the image feature points
anymore. To tackle this issue, we propose LF-VIO, a real-time VIO framework for
cameras with extremely large FoV. We leverage a three-dimensional vector with
unit length to represent feature points, and design a series of algorithms to
overcome this challenge. To address the scarcity of panoramic visual odometry
datasets with ground-truth location and pose, we present the PALVIO dataset,
collected with a Panoramic Annular Lens (PAL) system with an entire FoV of
360{\deg}x(40{\deg}-120{\deg}) and an IMU sensor. With a comprehensive variety
of experiments, the proposed LF-VIO is verified on both the established PALVIO
benchmark and a public fisheye camera dataset with a FoV of
360{\deg}x(0{\deg}-93.5{\deg}). LF-VIO outperforms state-of-the-art
visual-inertial-odometry methods. Our dataset and code are made publicly
available at https://github.com/flysoaryun/LF-VIO
