  Advances in computer vision and machine learning enable robots to perceive
their surroundings in powerful new ways, but these perception modules have
well-known fragilities. We consider the problem of synthesizing a safe
controller that is robust despite perception errors. The proposed method
constructs a state estimator based on Gaussian processes with input-dependent
noises. This estimator computes a high-confidence set for the actual state
given a perceived state. Then, a robust neural network controller is
synthesized that can provably handle the state uncertainty. Furthermore, an
adaptive sampling algorithm is proposed to jointly improve the estimator and
controller. Simulation experiments, including a realistic vision-based
lane-keeping example in CARLA, illustrate the promise of the proposed approach
in synthesizing robust controllers with deep-learning-based perception.
