  Simulating realistic sensors is a challenging part in data generation for
autonomous systems, often involving carefully handcrafted sensor design, scene
properties, and physics modeling. To alleviate this, we introduce a pipeline
for data-driven simulation of a realistic LiDAR sensor. We propose a model that
learns a mapping between RGB images and corresponding LiDAR features such as
raydrop or per-point intensities directly from real datasets. We show that our
model can learn to encode realistic effects such as dropped points on
transparent surfaces or high intensity returns on reflective materials. When
applied to naively raycasted point clouds provided by off-the-shelf simulator
software, our model enhances the data by predicting intensities and removing
points based on the scene's appearance to match a real LiDAR sensor. We use our
technique to learn models of two distinct LiDAR sensors and use them to improve
simulated LiDAR data accordingly. Through a sample task of vehicle
segmentation, we show that enhancing simulated point clouds with our technique
improves downstream task performance.
