  We propose SSA2D, a simple yet effective end-to-end deep network for
actor-action detection in videos. The existing methods take a top-down approach
based on region-proposals (RPN), where the action is estimated based on the
detected proposals followed by post-processing such as non-maximal suppression.
While effective in terms of performance, these methods pose limitations in
scalability for dense video scenes with a high memory requirement for thousands
of proposals. We propose to solve this problem from a different perspective
where we don't need any proposals. SSA2D is a unified network, which performs
pixel level joint actor-action detection in a single-shot, where every pixel of
the detected actor is assigned an action label. SSA2D has two main advantages:
1) It is a fully convolutional network which does not require any proposals and
post-processing making it memory as well as time efficient, 2) It is easily
scalable to dense video scenes as its memory requirement is independent of the
number of actors present in the scene. We evaluate the proposed method on the
Actor-Action dataset (A2D) and Video Object Relation (VidOR) dataset,
demonstrating its effectiveness in multiple actors and action detection in a
video. SSA2D is 11x faster during inference with comparable (sometimes better)
performance and fewer network parameters when compared with the prior works.
