  Retrieving accurate semantic information in challenging high dynamic range
(HDR) and high-speed conditions remains an open challenge for image-based
algorithms due to severe image degradations. Event cameras promise to address
these challenges since they feature a much higher dynamic range and are
resilient to motion blur. Nonetheless, semantic segmentation with event cameras
is still in its infancy which is chiefly due to the lack of high-quality,
labeled datasets. In this work, we introduce ESS (Event-based Semantic
Segmentation), which tackles this problem by directly transferring the semantic
segmentation task from existing labeled image datasets to unlabeled events via
unsupervised domain adaptation (UDA). Compared to existing UDA methods, our
approach aligns recurrent, motion-invariant event embeddings with image
embeddings. For this reason, our method neither requires video data nor
per-pixel alignment between images and events and, crucially, does not need to
hallucinate motion from still images. Additionally, we introduce DSEC-Semantic,
the first large-scale event-based dataset with fine-grained labels. We show
that using image labels alone, ESS outperforms existing UDA approaches, and
when combined with event labels, it even outperforms state-of-the-art
supervised approaches on both DDD17 and DSEC-Semantic. Finally, ESS is
general-purpose, which unlocks the vast amount of existing labeled image
datasets and paves the way for new and exciting research directions in new
fields previously inaccessible for event cameras.
