  Semantic segmentation networks are usually pre-trained once and not updated
during deployment. As a consequence, misclassifications commonly occur if the
distribution of the training data deviates from the one encountered during the
robot's operation. We propose to mitigate this problem by adapting the neural
network to the robot's environment during deployment, without any need for
external supervision. Leveraging complementary data representations, we
generate a supervision signal, by probabilistically accumulating consecutive 2D
semantic predictions in a volumetric 3D map. We then train the network on
renderings of the accumulated semantic map, effectively resolving ambiguities
and enforcing multi-view consistency through the 3D representation. In contrast
to scene adaptation methods, we aim to retain the previously-learned knowledge,
and therefore employ a continual learning experience replay strategy to adapt
the network. Through extensive experimental evaluation, we show successful
adaptation to real-world indoor scenes both on the ScanNet dataset and on
in-house data recorded with an RGB-D sensor. Our method increases the
segmentation accuracy on average by 9.9% compared to the fixed pre-trained
neural network, while retaining knowledge from the pre-training dataset.
