  Neural operations as convolutions, self-attention, and vector aggregation are
the go-to choices for recognizing short-range actions. However, they have three
limitations in modeling long-range activities. This paper presents PIC,
Permutation Invariant Convolution, a novel neural layer to model the temporal
structure of long-range activities. It has three desirable properties. i.
Unlike standard convolution, PIC is invariant to the temporal permutations of
features within its receptive field, qualifying it to model the weak temporal
structures. ii. Different from vector aggregation, PIC respects local
connectivity, enabling it to learn long-range temporal abstractions using
cascaded layers. iii. In contrast to self-attention, PIC uses shared weights,
making it more capable of detecting the most discriminant visual evidence
across long and noisy videos. We study the three properties of PIC and
demonstrate its effectiveness in recognizing the long-range activities of
Charades, Breakfast, and MultiThumos.
