  Existing action localization approaches adopt shallow temporal convolutional
networks (\ie, TCN) on 1D feature map extracted from video frames. In this
paper, we empirically find that stacking more conventional temporal convolution
layers actually deteriorates action classification performance, possibly
ascribing to that all channels of 1D feature map, which generally are highly
abstract and can be regarded as latent concepts, are excessively recombined in
temporal convolution. To address this issue, we introduce a novel concept-wise
temporal convolution (CTC) layer as an alternative to conventional temporal
convolution layer for training deeper action localization networks. Instead of
recombining latent concepts, CTC layer deploys a number of temporal filters to
each concept separately with shared filter parameters across concepts. Thus can
capture common temporal patterns of different concepts and significantly enrich
representation ability. Via stacking CTC layers, we proposed a deep
concept-wise temporal convolutional network (C-TCN), which boosts the
state-of-the-art action localization performance on THUMOS'14 from 42.8 to 52.1
in terms of mAP(\%), achieving a relative improvement of 21.7\%. Favorable
result is also obtained on ActivityNet.
