  We address the problem of temporal action localization in videos. We pose
action localization as a structured prediction over arbitrary-length temporal
windows, where each window is scored as the sum of frame-wise classification
scores. Additionally, our model classifies the start, middle, and end of each
action as separate components, allowing our system to explicitly model each
action's temporal evolution and take advantage of informative temporal
dependencies present in this structure. In this framework, we localize actions
by searching for the structured maximal sum, a problem for which we develop a
novel, provably-efficient algorithmic solution. The frame-wise classification
scores are computed using features from a deep Convolutional Neural Network
(CNN), which are trained end-to-end to directly optimize for a novel structured
objective. We evaluate our system on the THUMOS 14 action detection benchmark
and achieve competitive performance.
