  We present Recurrent Vision Transformers (RVTs), a novel backbone for object
detection with event cameras. Event cameras provide visual information with
sub-millisecond latency at a high-dynamic range and with strong robustness
against motion blur. These unique properties offer great potential for
low-latency object detection and tracking in time-critical scenarios. Prior
work in event-based vision has achieved outstanding detection performance but
at the cost of substantial inference time, typically beyond 40 milliseconds. By
revisiting the high-level design of recurrent vision backbones, we reduce
inference time by a factor of 6 while retaining similar performance. To achieve
this, we explore a multi-stage design that utilizes three key concepts in each
stage: First, a convolutional prior that can be regarded as a conditional
positional embedding. Second, local and dilated global self-attention for
spatial feature interaction. Third, recurrent temporal feature aggregation to
minimize latency while retaining temporal information. RVTs can be trained from
scratch to reach state-of-the-art performance on event-based object detection -
achieving an mAP of 47.2% on the Gen1 automotive dataset. At the same time,
RVTs offer fast inference (<12 ms on a T4 GPU) and favorable parameter
efficiency (5 times fewer than prior art). Our study brings new insights into
effective design choices that can be fruitful for research beyond event-based
vision.
