  Event cameras are neuromorphic vision sensors representing visual information
as sparse and asynchronous event streams. Most state-of-the-art event-based
methods project events into dense frames and process them with conventional
learning models. However, these approaches sacrifice the sparsity and high
temporal resolution of event data, resulting in a large model size and high
computational complexity. To fit the sparse nature of events and sufficiently
explore the relationship between them, we develop a novel attention-aware model
named Event Voxel Set Transformer (EVSTr) for spatiotemporal representation
learning on event streams. It first converts the event stream into voxel sets
and then hierarchically aggregates voxel features to obtain robust
representations. The core of EVSTr is an event voxel transformer encoder to
extract discriminative spatiotemporal features, which consists of two
well-designed components, including a Multi-Scale Neighbor Embedding Layer
(MNEL) for local information aggregation and a Voxel Self-Attention Layer
(VSAL) for global feature interactions. Enabling the network to incorporate a
long-range temporal structure, we introduce a segment modeling strategy to
learn motion patterns from a sequence of segmented voxel sets. We evaluate the
proposed model on two event-based recognition tasks: object classification and
action recognition. Comprehensive experiments show that EVSTr achieves
state-of-the-art performance while maintaining low model complexity.
Additionally, we present a new dataset (NeuroHAR) recorded in challenging
visual scenarios to complement the lack of real-world event-based datasets for
action recognition.
