  Event cameras detect changes in per-pixel intensity to generate asynchronous
`event streams'. They offer great potential for accurate semantic map retrieval
in real-time autonomous systems owing to their much higher temporal resolution
and high dynamic range (HDR) compared to conventional cameras. However,
existing implementations for event-based segmentation suffer from sub-optimal
performance since these temporally dense events only measure the varying
component of a visual signal, limiting their ability to encode dense spatial
context compared to frames. To address this issue, we propose a hybrid
end-to-end learning framework HALSIE, utilizing three key concepts to reduce
inference cost by up to $20\times$ versus prior art while retaining similar
performance: First, a simple and efficient cross-domain learning scheme to
extract complementary spatio-temporal embeddings from both frames and events.
Second, a specially designed dual-encoder scheme with Spiking Neural Network
(SNN) and Artificial Neural Network (ANN) branches to minimize latency while
retaining cross-domain feature aggregation. Third, a multi-scale cue mixer to
model rich representations of the fused embeddings. These qualities of HALSIE
allow for a very lightweight architecture achieving state-of-the-art
segmentation performance on DDD-17, MVSEC, and DSEC-Semantic datasets with up
to $33\times$ higher parameter efficiency and favorable inference cost (17.9mJ
per cycle). Our ablation study also brings new insights into effective design
choices that can prove beneficial for research across other vision tasks.
