  3D dense captioning stands as a cornerstone in achieving a comprehensive
understanding of 3D scenes through natural language. It has recently witnessed
remarkable achievements, particularly in indoor settings. However, the
exploration of 3D dense captioning in outdoor scenes is hindered by two major
challenges: 1) the domain gap between indoor and outdoor scenes, such as
dynamics and sparse visual inputs, makes it difficult to directly adapt
existing indoor methods; 2) the lack of data with comprehensive box-caption
pair annotations specifically tailored for outdoor scenes. To this end, we
introduce the new task of outdoor 3D dense captioning. As input, we assume a
LiDAR point cloud and a set of RGB images captured by the panoramic camera rig.
The expected output is a set of object boxes with captions. To tackle this
task, we propose the TOD3Cap network, which leverages the BEV representation to
generate object box proposals and integrates Relation Q-Former with
LLaMA-Adapter to generate rich captions for these objects. We also introduce
the TOD3Cap dataset, the largest one to our knowledge for 3D dense captioning
in outdoor scenes, which contains 2.3M descriptions of 64.3K outdoor objects
from 850 scenes. Notably, our TOD3Cap network can effectively localize and
caption 3D objects in outdoor scenes, which outperforms baseline methods by a
significant margin (+9.6 CiDEr@0.5IoU). Code, data, and models are publicly
available at https://github.com/jxbbb/TOD3Cap.
