  This work reports a novel Bundle Adjustment (BA) formulation using a
Reproducing Kernel Hilbert Space (RKHS) representation called RKHS-BA. The
proposed formulation is correspondence-free, enables the BA to use RGB-D/LiDAR
and semantic labels in the optimization directly, and provides a generalization
for the photometric loss function commonly used in direct methods. RKHS-BA can
incorporate appearance and semantic labels within a continuous spatial-semantic
functional representation that does not require optimization via image
pyramids. We demonstrate its applications in sliding-window odometry and global
LiDAR mapping, which show highly robust performance in extremely challenging
scenes and the best trade-off of generalization and accuracy.
