  Traditional video action detectors typically adopt the two-stage pipeline,
where a person detector is first employed to generate actor boxes and then 3D
RoIAlign is used to extract actor-specific features for classification. This
detection paradigm requires multi-stage training and inference, and cannot
capture context information outside the bounding box. Recently, a few
query-based action detectors are proposed to predict action instances in an
end-to-end manner. However, they still lack adaptability in feature sampling
and decoding, thus suffering from the issues of inferior performance or slower
convergence. In this paper, we propose a new one-stage sparse action detector,
termed STMixer. STMixer is based on two core designs. First, we present a
query-based adaptive feature sampling module, which endows our STMixer with the
flexibility of mining a set of discriminative features from the entire
spatiotemporal domain. Second, we devise a dual-branch feature mixing module,
which allows our STMixer to dynamically attend to and mix video features along
the spatial and the temporal dimension respectively for better feature
decoding. Coupling these two designs with a video backbone yields an efficient
end-to-end action detector. Without bells and whistles, our STMixer obtains the
state-of-the-art results on the datasets of AVA, UCF101-24, and JHMDB.
