  Roadside unit (RSU) can significantly improve the safety and robustness of
autonomous vehicles through Vehicle-to-Everything (V2X) communication.
Currently, the usage of a single RSU mainly focuses on real-time inference and
V2X collaboration, while neglecting the potential value of the high-quality
data collected by RSU sensors. Integrating the vast amounts of data from
numerous RSUs can provide a rich source of data for model training. However,
the absence of ground truth annotations and the difficulty of transmitting
enormous volumes of data are two inevitable barriers to fully exploiting this
hidden value. In this paper, we introduce FedRSU, an innovative federated
learning framework for self-supervised scene flow estimation. In FedRSU, we
present a recurrent self-supervision training paradigm, where for each RSU, the
scene flow prediction of points at every timestamp can be supervised by its
subsequent future multi-modality observation. Another key component of FedRSU
is federated learning, where multiple devices collaboratively train an ML model
while keeping the training data local and private. With the power of the
recurrent self-supervised learning paradigm, FL is able to leverage innumerable
underutilized data from RSU. To verify the FedRSU framework, we construct a
large-scale multi-modality dataset RSU-SF. The dataset consists of 17 RSU
clients, covering various scenarios, modalities, and sensor settings. Based on
RSU-SF, we show that FedRSU can greatly improve model performance in ITS and
provide a comprehensive benchmark under diverse FL scenarios. To the best of
our knowledge, we provide the first real-world LiDAR-camera multi-modal dataset
and benchmark for the FL community.
