  Video Analytics Software as a Service (VA SaaS) has been rapidly growing in
recent years. VA SaaS is typically accessed by users using a lightweight
client. Because the transmission bandwidth between the client and cloud is
usually limited and expensive, it brings great benefits to design cloud video
analysis algorithms with a limited data transmission requirement. Although
considerable research has been devoted to video analysis, to our best
knowledge, little of them has paid attention to the transmission bandwidth
limitation in SaaS. As the first attempt in this direction, this work
introduces a problem of few-frame action recognition, which aims at maintaining
high recognition accuracy, when accessing only a few frames during both
training and test. Unlike previous work that processed dense frames, we present
Temporal Sequence Distillation (TSD), which distills a long video sequence into
a very short one for transmission. By end-to-end training with 3D CNNs for
video action recognition, TSD learns a compact and discriminative temporal and
spatial representation of video frames. On Kinetics dataset, TSD+I3D typically
requires only 50\% of the number of frames compared to I3D, a state-of-the-art
video action recognition algorithm, to achieve almost the same accuracies. The
proposed TSD has three appealing advantages. Firstly, TSD has a lightweight
architecture and can be deployed in the client, eg. mobile devices, to produce
compressed representative frames to save transmission bandwidth. Secondly, TSD
significantly reduces the computations to run video action recognition with
compressed frames on the cloud, while maintaining high recognition accuracies.
Thirdly, TSD can be plugged in as a preprocessing module of any existing 3D
CNNs. Extensive experiments show the effectiveness and characteristics of TSD.
