  Frontier exploration and reinforcement learning have historically been used
to solve the problem of enabling many mobile robots to autonomously and
cooperatively explore complex surroundings. These methods need to keep an
internal global map for navigation, but they do not take into consideration the
high costs of communication and information sharing between robots. This study
offers CQLite, a novel distributed Q-learning technique designed to minimize
data communication overhead between robots while achieving rapid convergence
and thorough coverage in multi-robot exploration. The proposed CQLite method
uses ad hoc map merging, and selectively shares updated Q-values at recently
identified frontiers to significantly reduce communication costs. The
theoretical analysis of CQLite's convergence and efficiency, together with
extensive numerical verification on simulated indoor maps utilizing several
robots, demonstrates the method's novelty. With over 2x reductions in
computation and communication alongside improved mapping performance, CQLite
outperformed cutting-edge multi-robot exploration techniques like Rapidly
Exploring Random Trees and Deep Reinforcement Learning. Related codes are
open-sourced at \url{https://github.com/herolab-uga/cqlite}.
