  Transformers have recently been popular for learning and inference in the
spatial-temporal domain. However, their performance relies on storing and
applying attention to the feature tensor of each frame in video. Hence, their
space and time complexity increase linearly as the length of video grows, which
could be very costly for long videos. We propose a novel visual memory network
architecture for the learning and inference problem in the spatial-temporal
domain. We maintain a fixed set of memory slots in our memory network and
propose an algorithm based on Gumbel-Softmax to learn an adaptive strategy to
update this memory. Finally, this architecture is benchmarked on the video
object segmentation (VOS) and video prediction problems. We demonstrate that
our memory architecture achieves state-of-the-art results, outperforming
transformer-based methods on VOS and other recent methods on video prediction
while maintaining constant memory capacity independent of the sequence length.
