  Evaluating the performance of Simultaneous Localization and Mapping (SLAM)
algorithms is essential for scientists and users of robotic systems alike. But
there are a multitude different permutations of possible options of hardware
setups and algorithm configurations, as well as different datasets and
algorithms, such that it is infeasible to thoroughly compare SLAM systems
against the full state of the art. To solve that we present the SLAM Hive
Benchmarking Suite, which is able to analyze SLAM algorithms in thousands of
mapping runs, through its utilization of container technology and deployment in
the cloud. This paper presents the architecture and open source implementation
of SLAM Hive and compares it to existing efforts on SLAM evaluation. We perform
mapping runs of many of the most popular visual and LiDAR based SLAM algorithms
against commonly used datasets and show how SLAM Hive and then be used to
conveniently analyze the results against various aspects. Through this we
envision that SLAM Hive can become an essential tool for proper comparisons and
evaluations of SLAM algorithms and thus drive the scientific development in the
research on SLAM. The open source software as well as a demo to show the live
analysis of 100s of mapping runs can be found on our SLAM Hive website.
