  Different outdoor illumination conditions drastically alter the appearance of
urban scenes, and they can harm the performance of image-based robot perception
systems if not seen during training. Camera simulation provides a
cost-effective solution to create a large dataset of images captured under
different lighting conditions. Towards this goal, we propose LightSim, a neural
lighting camera simulation system that enables diverse, realistic, and
controllable data generation. LightSim automatically builds lighting-aware
digital twins at scale from collected raw sensor data and decomposes the scene
into dynamic actors and static background with accurate geometry, appearance,
and estimated scene lighting. These digital twins enable actor insertion,
modification, removal, and rendering from a new viewpoint, all in a
lighting-aware manner. LightSim then combines physically-based and learnable
deferred rendering to perform realistic relighting of modified scenes, such as
altering the sun location and modifying the shadows or changing the sun
brightness, producing spatially- and temporally-consistent camera videos. Our
experiments show that LightSim generates more realistic relighting results than
prior work. Importantly, training perception models on data generated by
LightSim can significantly improve their performance.
