  Camera-IMU (Inertial Measurement Unit) sensor fusion has been extensively
studied in recent decades. Numerous observability analysis and fusion schemes
for motion estimation with self-calibration have been presented. However, it
has been uncertain whether both camera and IMU intrinsic parameters are
observable under general motion. To answer this question, by using the Lie
derivatives, we first prove that for a rolling shutter (RS) camera-IMU system,
all intrinsic and extrinsic parameters, camera time offset, and readout time of
the RS camera, are observable with an unknown landmark. To our knowledge, we
are the first to present such a proof. Next, to validate this analysis and to
solve the drift issue of a structureless filter during standstills, we develop
a Keyframe-based Sliding Window Filter (KSWF) for odometry and
self-calibration, which works with a monocular RS camera or stereo RS cameras.
Though the keyframe concept is widely used in vision-based sensor fusion, to
our knowledge, KSWF is the first of its kind to support self-calibration. Our
simulation and real data tests have validated that it is possible to fully
calibrate the camera-IMU system using observations of opportunistic landmarks
under diverse motion. Real data tests confirmed previous allusions that keeping
landmarks in the state vector can remedy the drift in standstill, and showed
that the keyframe-based scheme is an alternative solution.
