  Maps have played an indispensable role in enabling safe and automated
driving. Although there have been many advances on different fronts ranging
from SLAM to semantics, building an actionable hierarchical semantic
representation of urban dynamic scenes and processing information from multiple
agents are still challenging problems. In this work, we present Collaborative
URBan Scene Graphs (CURB-SG) that enable higher-order reasoning and efficient
querying for many functions of automated driving. CURB-SG leverages panoptic
LiDAR data from multiple agents to build large-scale maps using an effective
graph-based collaborative SLAM approach that detects inter-agent loop closures.
To semantically decompose the obtained 3D map, we build a lane graph from the
paths of ego agents and their panoptic observations of other vehicles. Based on
the connectivity of the lane graph, we segregate the environment into
intersecting and non-intersecting road areas. Subsequently, we construct a
multi-layered scene graph that includes lane information, the position of
static landmarks and their assignment to certain map sections, other vehicles
observed by the ego agents, and the pose graph from SLAM including 3D panoptic
point clouds. We extensively evaluate CURB-SG in urban scenarios using a
photorealistic simulator. We release our code at
http://curb.cs.uni-freiburg.de.
