  Testing and debugging have become major obstacles for robot software
development, because of high system complexity and dynamic environments.
Standard, middleware-based data recording does not provide sufficient
information on internal computation and performance bottlenecks. Other existing
methods also target very specific problems and thus cannot be used for
multipurpose analysis. Moreover, they are not suitable for real-time
applications. In this paper, we present ros2_tracing, a collection of flexible
tracing tools and multipurpose instrumentation for ROS 2. It allows collecting
runtime execution information on real-time distributed systems, using the
low-overhead LTTng tracer. Tools also integrate tracing into the invaluable ROS
2 orchestration system and other usability tools. A message latency experiment
shows that the end-to-end message latency overhead, when enabling all ROS 2
instrumentation, is on average 0.0033 ms, which we believe is suitable for
production real-time systems. ROS 2 execution information obtained using
ros2_tracing can be combined with trace data from the operating system,
enabling a wider range of precise analyses, that help understand an application
execution, to find the cause of performance bottlenecks and other issues. The
source code is available at: https://github.com/ros2/ros2_tracing.
