  Autonomous vehicles are equipped with a multi-modal sensor setup to enable
the car to drive safely. The initial calibration of such perception sensors is
a highly matured topic and is routinely done in an automated factory
environment. However, an intriguing question arises on how to maintain the
calibration quality throughout the vehicle's operating duration. Another
challenge is to calibrate multiple sensors jointly to ensure no propagation of
systemic errors. In this paper, we propose CaLiCa, an end-to-end deep
self-calibration network which addresses the automatic calibration problem for
pinhole camera and Lidar. We jointly predict the camera intrinsic parameters
(focal length and distortion) as well as Lidar-Camera extrinsic parameters
(rotation and translation), by regressing feature correlation between the
camera image and the Lidar point cloud. The network is arranged in a
Siamese-twin structure to constrain the network features learning to a mutually
shared feature in both point cloud and camera (Lidar-camera constraint).
Evaluation using KITTI datasets shows that we achieve 0.154 {\deg} and 0.059 m
accuracy with a reprojection error of 0.028 pixel with a single-pass inference.
We also provide an ablative study of how our end-to-end learning architecture
offers lower terminal loss (21% decrease in rotation loss) compared to isolated
calibration
