  In this dissertation, I present my work towards exploring temporal
information for better video understanding. Specifically, I have worked on two
problems: action recognition and semantic segmentation. For action recognition,
I have proposed a framework, termed hidden two-stream networks, to learn an
optimal motion representation that does not require the computation of optical
flow. My framework alleviates several challenges faced in video classification,
such as learning motion representations, real-time inference, multi-framerate
handling, generalizability to unseen actions, etc. For semantic segmentation, I
have introduced a general framework that uses video prediction models to
synthesize new training samples. By scaling up the training dataset, my trained
models are more accurate and robust than previous models even without
modifications to the network architectures or objective functions. I believe
videos have much more potential to be mined, and temporal information is one of
the most important cues for machines to perceive the visual world better.
