  Robotics simulation plays an important role in the design, development, and
verification and validation of robotic systems. Recent studies have shown that
simulation may be used as a cheaper, safer, and more reliable alternative to
manual, and widely used, process of field testing. This is particularly
important in the context of continuous integration pipelines, where integrated
automated testing is key to reducing costs while maintaining system safety.
However, simulation and automated testing are not seeing the degree of
widespread adoption in practice that their potential would motivate. Our goal
in this paper is to develop a principled understanding of the ways developers
use simulation in their process, and the challenges they face in doing so. This
type of understanding can guide the development of more effective simulators
and testing techniques for modern robotics development.
  To that end, we conduct a survey of 82 robotics developers from a diversity
of backgrounds that addresses the current capabilities and limits of simulation
technology in practice. We find that simulation is used by 85% of our
participants for testing, and that many participants desire to use simulation
as part of their test automation. We identify 10 high-level challenges that
impede developers from using simulation for manual and automated testing, and
general purposes. These challenges include the gap between simulation and
reality, a lack of reproducibility, and considerable resource costs associated
with using simulators. Finally, we outline avenues for improvement in the
development of new simulators that can help simulation reach its potential as a
means of verification and validation.
