  Event-based vision represents a paradigm shift in how vision information is
captured and processed. By only responding to dynamic intensity changes in the
scene, event-based sensing produces far less data than conventional frame-based
cameras, promising to springboard a new generation of high-speed, low-power
machines for edge intelligence. However, processing such dynamically sparse
input originated from event cameras efficiently in real time, particularly with
complex deep neural networks (DNN), remains a formidable challenge. Existing
solutions that employ GPUs and other frame-based DNN accelerators often
struggle to efficiently process the dynamically sparse event data, missing the
opportunities to improve processing efficiency with sparse data. To address
this, we propose ESDA, a composable dynamic sparse dataflow architecture that
allows customized DNN accelerators to be constructed rapidly on FPGAs for
event-based vision tasks. ESDA is a modular system that is composed of a set of
parametrizable modules for each network layer type. These modules share a
uniform sparse token-feature interface and can be connected easily to compose
an all-on-chip dataflow accelerator on FPGA for each network model. To fully
exploit the intrinsic sparsity in event data, ESDA incorporates the use of
submanifold sparse convolutions that largely enhance the activation sparsity
throughout the layers while simplifying hardware implementation. Finally, a
network architecture and hardware implementation co-optimizing framework that
allows tradeoffs between accuracy and performance is also presented.
Experimental results demonstrate that when compared with existing GPU and
hardware-accelerated solutions, ESDA achieves substantial speedup and
improvement in energy efficiency across different applications, and it allows
much wider design space for real-world deployments.
