  Experience and reasoning occur across multiple temporal scales: milliseconds,
seconds, hours or days. The vast majority of computer vision research, however,
still focuses on individual images or short videos lasting only a few seconds.
This is because handling longer videos require more scalable approaches even to
process them. In this work, we propose a framework enabling research on
hour-long videos with the same hardware that can now process second-long
videos. We replace standard video compression, e.g. JPEG, with neural
compression and show that we can directly feed compressed videos as inputs to
regular video networks. Operating on compressed videos improves efficiency at
all pipeline levels -- data transfer, speed and memory -- making it possible to
train models faster and on much longer videos. Processing compressed signals
has, however, the downside of precluding standard augmentation techniques if
done naively. We address that by introducing a small network that can apply
transformations to latent codes corresponding to commonly used augmentations in
the original video space. We demonstrate that with our compressed vision
pipeline, we can train video models more efficiently on popular benchmarks such
as Kinetics600 and COIN. We also perform proof-of-concept experiments with new
tasks defined over hour-long videos at standard frame rates. Processing such
long videos is impossible without using compressed representation.
